If it is a previous workplace, from the way you phrased your question I'm gathering you don't see the people who work there on a regular basis anymore. Therefore I wouldn't worry too much about offending people. If you no longer see them, they won't be too offended as a) they are probably not inviting you to their parties either and b) you won't see them so you'll avoid all embarrassment anyway.
If you do still see everyone, can you not just have a private word in the ear of those who are going to your party?