Spanish and English relationship

I am currently living in Spain and have a good relationship with a wonderful Spanish man (I am english). We are of similar age and careers. We are both divorced and have grown children -mine is 27 his are 20 and 17. However, he still shares the family home with his ex wife, she lives there two weeks and he lives there two weeks. BUT, she moves in with her boyfriend for the two weeks she is not looking after the children. HE stays in the house in the basement all the time! AND I am not allowed to stay there!! I have tried to discuss it with him but her refused to change his mind!? He says he will in the future but not say when. Because of this, I have taken a job in Thailand until he makes himself freer!? He says he will visit me. He gives me everything I want, he spends all his free time with me. I dont feel our relationsip can grow naturally eg our sex life suffers a we have to go hotels and friends and i dont like it! I dont know if i should give this relationship time - as he has only recently divorced OR finish? Advise welcome please????