Problem with my friend?

WARNING: This is REALLY long and if you really want to help me you can read it, thank you very much! ;) ------------------------------------------------------------------------------------------------------- So me and my brother have a friend and he's really friendly (sometimes) and all, and we know each other for a long time since our childhood and me and my brother still see him everyday and we're friends you know but if my brother is not there and me and my friends are alone, he speaks about my brother? The same goes for my brother, if my brother is alone with him he speaks behind my back. Today after fitness I was alone with him (we went together) and he told me that my brother can't fight, he always does this *shows the move* (He did 2 years of Thaibox that's why) and he then shows a video of my brother fighting our friend just sparring for fun you know and he told me I'll let you see the move, but the move my brother did was not in the video and so I kept silent and said 'yea, you did 2 years of Thai so you're better at techniques than us, but we're better at power, techniques are our weaknesses so..' and he said yes I know. After that I told him that my brother will maybe do fitness and then Martial Arts and he said 'what? I thought your bro is a basketballer' and I told him yes I know he is maybe going to do the two at the same time and he told me your bro doesn't have a will to do basketball, I mean like he didn't go yesterday, he said. I told him yes it was for personal reasons and I told him the reason which was something and then he said that before I had the same problem too and I still went to the gym and Thaibox. After that we were still walking and we were talking normally and then he told me you never have the will to do your best or to do something, I told him you're underestimating me he told me no I'm not. I then told him when we play outside do you really think that we do our best? He said yes you often do your best and then I told him then you don't know much about us. And before we went to the gym when we were walking he said you are not the kind of guy who would fight if he doesn't have to, you're friendly and you talk it out. And he meant that for my brother too (Both me and my brother are friendly he said). So.. after I went home I told my brother that he talked about you and he said, really? I said yes and he then said he does talk about you too when we're alone. So.. We've busted him a couple of times lying about somethings and it is NOT the first time he did this so, can you tell me what to do? I don't want to talk it out about him. Should I just ignore what he says about us and keep going?