Should I leave him

This is long so bear with me: Me and my sons father have been togeather for 3yrs. We didn't exactly meet under the best circumstance, and quickly found ourselves pregnant. (Yes, he's truly the father...) he is a wonderful guy. Even though we weren't dating he stepped up and decided to be there and help raise the child.. well we moved away during the pregnancy and 1st year he was born. We figured it was better for all of us if we just threw ourselves in a remote location, only depending on each other. But like all things, it came to a end. We moved back after I found out my father was sick. (During the time we were gone I had lost 3 family members, and couldn't afford to go to the funerals..so we moved back.) But we were making far less money here then when we were living in Fl. We ended up moving his father inside our house, to watch our son while we both worked. But that didn't go over well... his father is a repeat addict, who tricked us into bringing his dope friends into our house. Only for them to squat in our home, and physically chased us out of our own home.... For most of the arguement he actually took his fathers side (thinking that I was mad because they stopped lending me stuff.) Which was entirely not the case... it was the fact that they were doing drugs and drinking their asses of while watching my child! Which granted he wasn't home the majority of the time anyway.... I'm getting off topic. Anyway, I have to say that's when it first started happening. That's when I first stopped having emotional feelings for him... he is always away from home... he wakes up at 10am goes to the gym as soon as he wakes up.. doesn't come home until 1pm. Then he gets in the shower, eats lunch, and off to work he goes at 3pm... I never get to see him. Which I could deal with mon-fri. But even on the weekends, as soon as he wakes up off he goes. He wants to go play disc golf one day, or wants to go skateboarding the next. Always wants to go to the movies. But fails to realize that we can't do all those things... we have a child to look after, and very little money to spare. Yet all I hear is let's do this, or let's buy that. But we can't, so then he gets mad, and pouts and goes on until he gets to do it anyway... He does take our son 2 the gym with him (but they have a daycare.) He never physically plays with our child, at least not for a long period of time.. and it actually makes me sad, because every time daddy leave; our son is banging on the door yelling for him not too.. I just feel like we are on the back burner in his prioritys.... is it wrong of me to want to leave him? We were never even togeather in the 1st place really; just to people who got in over their heads trying to do what was right.... But I just can't be happy anymore.. I can't be content spending all day, everyday alone.. and when I say that, I mean alone even though I shouldn't be... I understand I will be alone if I leave; but in a way I would feel so much more happier.. I'm just confused I guess. On what's the right thing for me to do for myself as well as my son, and even for his father... because all of our lives would be thrown into chaos... anyway, any advice is welcome.