Lost and confused, need help

My boyfriend, well now my ex boyfriend and I were together about a year and a half. We were in love and he asked me to marry him, I accepted. We live in different states and he was just in town to visit me and had left on Sept. 22nd. On, Sept. 23rd, Iwas at my sisters house which is two hours away. Everyone else was out or on their way back to her house. My brother in law and I were at my sisters alone. He walks into the living room where I was sitting, pulls down his pants, his underware grabs my ches (all in one instant) and then says to me, .."lets do it...wannna do it". I blocked myself and pushed his arms away and when my sister got back with my daughter, we left. Before they got back he said either "don't be freaked out or don't be all freaked out" and asked me if I was going to tell my sister. They were going to divorce last I saw them. He is still there and last night things seemed pretty ok..so I guess those plans were on the back burner... When my sister gets back with my daughter, I get my stuff and head home. I didn't tell my sister yet. I was so upset and didn't want to be MORE shook up for a two hour drive home and scare my daughter with the confrontation. So on the way home I call my boyfriend and tell him what happened. He is upset and scared for me. I was not physically hurt. He is furious at what my brother in law did and then starts the questions. He didn't like I handled it...and so this is what he had to say... He was MAD, overall. Then he was going on and on about why I didn't tell her right then and there and told me "you won't listen to me, you are going to do what you are going to do" Asked again if I was ok and then said "You have an exciting life, no guy ever comes over and drops his pants in front of me"....that part was supposed to be funny in his lame attempt to lighten the mood, but I find it HORRIBLY INSENSITVE and cruel. He was finishing up at work and told me to call him when I got closer to home. I did just th atat 8:30 p m. on Sept. the 23rd. I also mentioned to him about calling me back when he says he is going to....he got bent out of shape and said he isn't going to argue about that...said he would talk to me later, and bye....he then hung up. That was at 8:30 p m on Sept. 23rd. The boyfriend ended up calling back the next day at 6:00 pm on his way home from work to see how I was feeling. I don't know, but I was pretty upset that I had not heard from him since earlier the night before. It wasn't like I broke a toe, I was sexually assaulted. When he did call, I asked him why he didn't call earlier, even to see how I was....he said that he didn't want people to hear him at work and start asking all kinds of questions. I guess stepping outside or telling them "none of your business" is out of the question. Overall I wasn't happy with his attitude and I told him "You dropped the ball buddy." He replied with "Buddy????" and then said he didn't want to say the wrong thing or have me take something he said the wrong way, he didn't want to hurt me. So he said he would talk to me later. That was Sept. the 24th at 6 pm, the day AFTER the assault. I have not heard from him in any way since (except in a short email on my bday that only said "happy birthday". I'm sorry, if the love of my life had been sexually assaulted or abused, no busy day would keep me from checking on them.... Is the boyfriend being selfish? Was I out of line? Do I expect too much? So this is where the statement comes in..... On thanksgiving night I sent him a text message saying I hope you had a good turkey day and am glad you are ok (a generalized ok). He replied back via text with “I’m hanging in there”. I told him to stay safe. He replied with “U 2 with your new luv”. I don’t have a new love. He said he had read my posts and comments on a forum we are both on, but he took he misinterpreted what it meant. He thought I had a new love. I don’t. What it meant was I found love, within myself. I clarified that with him. He can be a jealous guy. I then asked him if he had someone new (he replied with no, after he made the statement below) THAT’S when he said about himself “I am not a person to be loved by anyone”. Can anyone explain that statement to me?????