How do I break up with him?

hi, i am 20 years old, i have been with my partner for 3 years, we have a 9 month old son. For the past year my partner has been paranoid that I have cheated on him (I havent) he was also out of work for most of my pregnancy and worked up until 2 wks b4 my due date to try and make ends meet. I lived with my grandmother at the time and her support was fantastic. My partner called me repulsive to look at when I was pregnant, which has damged my confidence, we moved in together a few months ago things where a little better until I opened a letter from the council informing me that we were in nearly £2000 arrears and where due in court the following week, my partner had known for sometime but had not told me. I put everything I had into our first home, I now have nothing and am in debt with a ccj at 20! he has not payed a penny towards our sons upkeep, my son and i are living back with my gran. I want to break up with my partner, I am afraid of his family though as they have a reputation in the town where I live and I do not want to bring trouble to my grans door, does anyone have any advice on what I should do?