How to let go when is the right time?

Hello, I don't really know where to begin. I am tangled in a web that's crazy in my life. I'm lost and confused. I know I have done it to myself and don't know how to fix everything. So this unfortunately will be very long, Hope you don't mind. Well Beginning to end.... I'm divorced once and my husband is divorced once. We met on Myspace on Sept 9th 2010. He just turned 41, I turn 25 next month. He has two children a girl (15), and a boy (13) with whom he has full custody of. I have a daughter (9) in which i have full custody. We married 7/2/2011. I love him so very much but I have been depressed lately hes been controlling and I felt like I didn't wanna fight for us anymore. Our sex life has decreased our fighting has increased. We fight because I don't clean to his expectations. or if i'm not disciplining my child as much as he wants me to. All the while his kids don't help at all around the house nor do they have any rules. His children stay up all hours of the night, eat sodas and candy for breakfast, Curse, talk back, get bad grades, they even steal. I have tried to give rules and hand out chores but I get undermined. He screams when my daughter forgets to clean her room but never asks his children to clean theirs. I sit back n stay quiet its hard to not love him even after all this. When we argue i kind of shut off i stay silent n just take it because i feel as though everything i say i don't mean bad but its interpreted as bad, and when he gets made he reminds me of the fact the house is his and his moms and tells me to pack my things and go. His oldest befriended all of his ex girlfriends on facebook and also goes to their houses to hang out she says I'm stupid and i disrespect her in her own house. She gives me dirty looks and we haven't even officially spoken in a month. Well now my side I haven't exactly been 100% in my marriage though either. I have been having an affair for the last 2 months it was nice to have someone tell me I was beautiful and buy me flowers and candy. My husband wont unless i say im fat i get "you're fine". I have received only one set of flowers since we have been together. I feel horrible but cant ended it because the guy i had an affair with said he loves me and he will tell if I wont leave my husband so I continue. Now he wants us to move in together and I dont think I'm ready but I know I cant say no. I know if my husband knew he would kick me out anyways and how could i not blame him. I know this because if he is 100% willing to kick me out for not cleaning properly then i cant expect him to be like ok this is fine. I have more of a reason to leave him then stay anyways but i still have love for him i dont know why i wish i did but its not so easy. I cant afford to move out on my own and I want to just tell my husband and hope he would understand why I did what I did but i realize that is unrealistic. What do i do? How do I let go? I want him to just be like get out of my house and finally i can go... what would you do?