Am I destined to be single?

I was married for a long time to my first wife, had children and then left because I was unhappy. Found second wife and now after a shorter time than expected find that once again I am unhappy. Should I walk away and just stay single and not muck up anybody else's lives as that is what I just seem to end up doing?