Advice needed please

So me and my husband for a about four years have been discussing swinging. Him more so than me. He wants to have threesomes with another male involved of for me to be with another female. I was against it for so long as it felt like if I did it I was cheating on him. Anyway eventually he persuaded me to try it once. So we did and it wasn't so bad. He kept telling me to message men and if I wanted to I could meet them alone. I did not take him up on the offer but thought about it. Anyway a few months later he kept mentioning that my female friend fancied me And it was obvious. I couldn't see it to be honest so I brushed it off. He kept going on about kissing her and I should have sex with her. One night I got drunk and we did stuff together while her boyfriend was having sex with her but he did not touch me. The next day before I could tell him about it he told me that he didn't like her or her partner anymore. I haven't told him about it because I didn't want to make him angry or upset with me. Now I feel like I have cheated on him and I couldn't bare to lose him. I've now left it really late to tell him about it. Now I don't know what is normal for our relationship. If I tell him I'm scared that he's going to leave me and we have a family together. We are also married and have a business together. We have been talking about having another baby soon and it's what I want more than anything. I know I have to tell him but I'm scared of what the outcome will be. He will either leave me and make a huge thing out of it (he shouts and breaks things and thinks about what he's done and said later) or he will be a bit annoyed but be okay with it. Can anyone please give me some advice please? I'm really stuggling at the moment with it so I would appreciate non judgmental answers please x