I don’t want him to think I am going there only to see him and that this is a way of stalking him or making him take me back but also I want to enjoy seeing my old colleagues with which I stayed in contact after I left my workplace.
See these former workplace people in another venue.
Go to the oarty. Act as if he doesn’t matter. He could skip the party himself, who knows?
It's probably best to focus on everyone else there and not your ex, though. I'm not saying you even have to avoid the ex, but you know you're going there to see all of these friends together at once again.