Heartbroken and confused

So, I met this great guy a couple of months ago. felt like I could not be more in love and feel more special. loving messages, spending nights together (almost all of them!!) Then slowly he started pulling away, he is looking for work, he needs to get his child back and pushing me away. when I try and talk it doesn't work, he wont really talk to me. if I overreact he tells me not to worry (as in getting jealous or insecure) and I just want to know I matter. We were going to move away together, now he says he doesn't want to move in, I realize its too soon. now he is planning on getting a place near here (I am still moving farther away) with his child as soon as possible. don't get me wrong, of course I know children matter I have my own as well. The more I try n find a connection, have faith, believe in love as long as you stay and work through things, he does not. The other night I went over, he wasn't cuddly, before 6 am we were both tossing and turning. I said what is going on? are you seeing someone else? are you trying to? he said no. and its too soon for us to move in together. which I knew. and he wants to focus on his kid. and I said I thought we had something special, that we were in love. he said its not love. I asked him what that meant. you don't love me? no answer. I said - do you want to break up? he said its up to you. and just layed there. I am such a fool, I said I would stay, and accept not living together. he went to the next town that day so didn't get to reconnect or whatever. now he told me he is looking for a place with his son as soon as this weekend. I am busy packing my own house to stay with my relatives until I move to the city in a months time. my heart is breaking. this hurts. I should just break up with him. I don't know if I can or want to. I love him so. despite this confusion, making it worse myself, still talking to my ex and he is being so nice, we just message I haven't seen him in person for quite awhile, been broken up over a year, topsy turvy 4 years together. help!!