As is so often the case, Terry Pratchett has the answer and its not (entirely) based on skin colour
"They called themselves the Munrungs. It meant The People, or The True Human Beings.
It's what most people call themselves, to begin with. And then one day the tribe meets some other People or, if it's not been a good day, The Enemy. If only they'd think up a name like Some More True Human Beings, it'd save a lot of trouble later on."
Terry Pratchett The Carpet People. When you think he originated this thought at around the age of 17......I mean what a brain.