The line is drawn, perhaps, when you say that you cannot stand people in group X, when a person from that same group is listening. If after they challenge you on it you still insist on saying the same thing, the line has been well and truly crossed. So, yes, if I said "I hate Frenchies" and a Frenchman is listening, he might well be offended by that.
That's not to say I shouldn't say it when I happen to know that there aren't French people nearby -- but certainly in the case when someone who is listening might be offended, you should jolly well watch what you say.