Just think about the fact that your bosses attitude is to compensate for other inadequacies (sp?). I'd lay money on it that your boss probably has a tiny willy, and the stamina of a shrew. In fact his wife probably laughs at him whilst she is with the milkman.
As for your colleagues, screw 'em. If they don't have the balls to bitch about you to your face but instead abuse your good nature, then they're not worth wasting brain power over. Just rest assured that they're probably bitching about each other to the rest of the office as well.