Dating and marrying and, heck, associating across races can still be difficult at times to this day. Even if you live in fairly diverse place, you can still experience a lot of these problems. And I think it can also be scary because to some degree any of us are capable of making these generalizations or assumptions without really sitting down and thinking about them. And some people convince themselves of one reason because the truth isn't as neat or easy to understand or believe, or because their environment convinces them of something.
It's hard enough working out differences with your partner as it is, and these outside forces weighing in with their own judgments just makes it so much more tough.
But at the same time, you love this woman. You made the choice to be part of one another's lives despite your differences, and to try to make things work. And maybe it's worth putting up with these problems and finding a way to shrug them off or deflect them for her and your family.
You know, you're under some pressure. Your baby girl grew up and left, and you miss her. You worked full time to support your family, and it feels like all of that work should have gotten you something more peaceful than this. Your parents were out of the picture from a young age, and the father who did turn up was a nightmare. You feel kind of different from everyone else and that bothers you. There's nothing wrong with having a few drinks and trying to loosen up if that helps take the edge off! But getting angry drunk and going fighting-round-the-block isn't going to make things better.
Try to let go of the past as much as you can, and work towards building a future free of that pain. You may not be able to fix everyone else's perceptions, but you can focus on improving your own mindset, and removing the negativity from your life that still holds you back.