AOG: "Is having a black skin and any association with Africa, something to be ashamed about?"
Historically people having those specific characteristics were subject to somewhat more than name calling, it meant a lower social status and less rights. This is why direct allusions to those specific characteristics by comparison to monkeys for instance is just more than simply "name calling", it's referring to the fact that the recipient is of a lower social status than the person saying it - this is not implied when referring to slap heads or ginger nuts.
Football chants about someones dead brother for instance is on the face of it just words, but to say "sticks and stones.." won't stop that person feeling negative emotions.
Comparisons to monkeys for other characteristics is more acceptable however but to address your original point it's skating on thin ice using it to describe black people because of the other connotations, even if you just meant they have low intelligence, or even that they are excellent at climbing trees.