Based on what you have said, you have my sympathy. I have learnt that one's conscious mind merely decides the gist of what one wishes to convey, it is the subconscious 'subroutines' that grab known words/phrases to form the actual speech (otherwise we'd all stumble searching for each word in turn). If your life means a phrase is well known to mean something specific that differs from a literal translation, one that in the meanwhile society has chosen to take offence to, then this sort of unintended offence is something that can make any of us its victim.
Your tale seems plausible to me. And the real issue appears to be either folk judging not understanding, or doing so yet deciding they dare not appear to be giving tacit approval for racism from folk knowing they could get away with it by claiming it was accidental and not meant. (Or maybe simply not believing you. )