Coping with relationship anxiety

My boyfriend and I started dating my junior year and his senior year in high school. The anxiety started small, that I was wanting to look at his phone and his facebook without any reason. And I found out that he had been lying about harmless things that didn't matter. His friends hung out and there was a girl there, he went out and said he fell asleep early, etc. Nothing incriminating, it was just to avoid an uncomfortable conversation. But that pretty much became the basis of my anxiety, and it got worse. Constantly worrying that he was lying, trying to find out if he way, getting more and more upset every time he did. After dating for 4 years, and him being 22, he wanted to go out with his friends to the bar and drink. He pretty much wanted the college experience and, with my anxiety, it made it unbearable for both of us. We broke up for about 5 months and it was honestly great for the both of us. I dated some guys for a short amount of time and never had anxiety with any of them, or had anxiety in other aspects of my life. Two months ago, the boyfriend and I got back together. My anxiety is more controlled now. And by that I mean I can handle it better, but it is still there. I think of something out of nowhere, with no basis, and I convince myself it's true to the point that I panic over it. Now, I can usually talk myself out of it instead of calling him in the middle of the night. Or, when that doesn't work, I can ask him nicely and in an accusatory way to relieve my worries instead of starting arguments with him. Still, those panic feelings are awful, even if I can push them aside easier. And they happen all the time, I might have one every other day, but only have to confront him about something once a week. It gets really bad when he goes out drinking with his friends, because I link that to the lying and the rejection from around our breakup. But obviously I can't tell him not to go out with his friends and I don't want him to stop, I just want to stop feeling so awful about it. He says that he still loves me and doesn't mind it, but I just feel like me being like this can't be something that someone can actually enjoy being around. And that it's just a matter of time until I drive him away again. I know that's a self-fulfilling prophecy, but I honestly can't control it. I don't know if I should just leave this relationship all together or if that is the easy way out and I need to stop being crazy.

Sorry, I should add that since we've gotten back together I have not looked at his phone at all. And he is a lot better about nicely answering my questions instead of getting defensive and angry like he used to. It really is better, but I don't know if better is good enough. Also, thank you for reading and responding, Raina