Boyfriend puts son before me, overreacting or not?

Hi All, My boyfriend and I have been together for 4 years. He is divorced and has 2 children,age 17 and 22 from his previous marriage. I get along with his youngest boy but the older one wants nothing to do with me which hurts me a lot. The reason for this is because he sees me as the reason for his parents divorce, it was a nasty divorce in which my name was slandered a lot. My boyfriend has his own house and I have mine,so whenever his oldest son wants to visit his father I go to my house. This has been the case for the last 4 years. Whenever I bring up the fact that I dont like the way his son is handeling the situation, my boyfriend always says that It's also hard for him as he is in the middle of all this and doesnt want to hurt either one of us and that he wants to restore the bond with his son. A couple days ago me and my boyfriend had a serious argument which left me feeling sad and not understood. His son texted him in the middle of us trying to work things out and wanted to come over and asked him what he was doing , my boyfriend response was 'Nothing'. (Basically giving him permission to come over. I asked my bf why he didn't just tell him he was with me,as we were trying 2 resolve our argument Instead of now expecting me to make room for this son. He told me that if hè had said I was there his son wouldnt come over. I packed my stuff and left in tears, feeling sad and worthless. It feels as if hè just doesn't care about my feelings. Like im not a priority. I feel like he should've picked me this time and not let his son come over. I dont know if Im wrong about this? Am I overreacting, should I accept this behaviour? I understand that that is his child and children always come first but right now I feel really shitty. Am I wrong? Sorry if my English is bad, Im from Belgium.