Cat and boyfriend don't get along

Seriously. This is my problem. My boyfriend and I adopted our cat a few months ago, and the two of them have never gotten along. My boyfriend is not terribly tolerant of the cat's quirks (bad) behaviour, and thinks that punishing the cat is going to help. Clearly it hasn't, since the cat is afraid of him now and won't warm up to him AT ALL. And the cat disliking him makes him dislike the cat even more. I get along great with the cat, and I really don't want to get rid of him, but I don't think it's fair to my boyfriend for us to have a pet that he doesn't get any enjoyment out of. I don't know what to do. Advice?