Very possibly they did -- on the other hand, that whole view died some years ago. Racism is still prevalent even today.
It's not difficult, really. If a white man used a racially-charged term about black men, it's not at all obvious if they are racist or just joking. If a black man makes some racist comment about black men, you can be confident that they're joking.
And it does, or should, work the other way too. That it does not is simply because we haven't quite shaken free the world of rampant racism, and sexim, and ageism and all the other -isms.