Is it okay to put a family member to a mental institution for the rest

he has mental illness and he is living with us and our family are toxic now because of him we are all stress by him..we want him out of our life..

Yes, sometimes it's the done thing to do particularly if you and yours are suffering and genuinely trying to cope with it. You can't help someone when you're beaten down trying to do just that. Some people are better off in institutions, while others go backwards. It's your call, and don't forget the legalities of it all as well.