In a mess

Please give me honest opinions. I was with my first Husband for 14 years. We had two lovely children. We were a close unit for many years but grew apart as the tiredness from working full time jobs with small children and little time together took their toll. He could also be very moody if things weren't his way. He ended up having an affair. We stayed together to try but it was difficult and I found out he was still in contact with the girl- buying us the same Christmas present. I found out he was on sex contact websites etc. It was too much so I called it a day. Three months after we split I met a man who became my second Husband a year later. I was foolish on the rebound and dived in too quickly missing that family feeling and not really fitting in. He treated myself and my children very poorly (including his Son). I finished that relationship for the sake of my children. I came away from my second marriage with three children reliant on me with no help my second Husband didn't bother with his child at all. I lost everything due to my second relationship as my Husband cheated and lied(He refused to contribute financially so I lost my house, job and had to start again). The children are my world. The children and I now finally have a house which I worked hard to save for and we are settled. My first Husband has always been around as we have joint custody. He got married a second time also but that ended in divorce as his wife was an alcoholic. He's made it clear he wants us to get back together. I care about him but I am scared by relationships. I am scared of altering the children and my life unless its for the better. He can be a kind man if you are doing what he wants but can also be very controlling when he comes to my house he tries to say what we watch etc. He can also manhandle our 12 year old Son which I don't agree with as he has anxiety and can be hard to deal with but I don't agree and I'm concerned he may do that with my youngest who only has me. We have gone out with the kids over the past year but I wont go further than friends as I'm scared if its what I really want and if I'm going to make things worse for the kids. He can be extremely bitter and in the past when I was seeing someone else he used call me a bitch and a slag if the kids didn't come on time when he turned up etc. He also became very bitter and wouldn't let the kids come over if they wanted to. I like living on my own with the kids I feel at peace and they enjoy our time. I feel worried about how they are when they are with him and they say they miss me and I miss them so I do go out with them with their Dad. My oldest boy wanted to come and stay with me Sunday night he stayed at his dads the night before and because he did that his Dad wouldn't take him to school. I hate the kids being made to feel that way and I think would it better if I was with him for them?. I had a long chat with him last night he said he's got three women he's texting/seeing with at the moment as he is single (which he is) and he's fed up waiting for me to show him some affection. I feel very confused as to what's the right thing to do. He said last night my youngest son (Who isn't his) is a reminder of the past. He does like him but its difficult. I don't ever want my youngest to be treated differently or made aware of that because his Dad doesn't bother with him already and that's hard enough for him. I've never seen him treat him differently but would it be different if we were together knowing he thinks like that? My plan was to be on my own until the kids are older as they have been through enough. I don't know if I'm holding on because he was a major part of my life or I do miss being a family and because I'm scared I am going to miss out on an opportunity?

Look I agree with Susiedqqq...DON"T go back to that person..ever! You and your kids seems to be doing wonderful without any of these abusive controlling men...keep on enjoying your time with yourself and your kids! These bad men made you lose EVERYTHING before and now you have it all back,...WITHOUT their help.Please learn to enjoy positive things in your life.You worked very hard to get where you are now.Why lose it again?