Joining families

My boyfriend and I have been dating for exactly 1 year now. In the beginning it was a long distant relationship. We get along great and I think he is a very considerate person. We both have gone through divorces on marriages that where he was married for 8 years and I was married for 12. He has 3 kids and I have 2. We dated for many months before seeing each others kids. It was 6 months before he meet my kids and 10 months before I meet his. Eventually I brought my kids to meet him but I had not meet his kids yet. As time past he ended up moving back into a home he shared with his wife and kids. Within 6 months of our relationship it got really serious and we talked about getting married and possibly having 1 more child. He asked me continuously to move to his city which is about 4 hours from my hometown. I thought the relationship was great and we where talking about putting our families together and thought it was a great idea. So I moved my kids and myself to his city. Now that we are here things has drastically changed. Therre is no more entertainment and he does nothing but work. The kids hardly ever get together nor him and I. The situation is he is now living in his old home that he shared with exwife and kids and he has a serious problem with me and my kids being there. If his kids are there we cannot spend the night. He said its because this is the home he shared with his wife and kids. But at the same time evrytime I invit him to go to my hometown with his kids and says he cannot do that because his kids are not ready. Meaning he cant imagine going overnight together with the kids.and having to spend the night together. Mind you his ex wife lives with another man with these kids. Im really caught and stuck on the fact that we plan to get married but I can hardly be around his kids or any of his other family. His family lives about 2 hours away and he takes his kids and visits for the weekend and me and my kids or never invited. It came down to my kids where away for the weekend and he ended up getting his and we went out for the night and arrived back late. I asked him if it was ok if we slept together and he said yes but after we where in bed he said I think Im gonna sleep upstairs. It was a slap in the face until it made me leave and go home at 2 am in the morning. Mind you, he sleeps at my home at least once a week with my kids there. Before I moved he slept in my home that I shared with my ex husband. My kids where present also. I feel like we have a commited relationship and future plans of marriage there should not be a problem. I think he needs to sell or lease out the home if he has a problem with us being there.