I've always thought that it was just that black sheep are a minority and the person is a bit unusual. It has to be admitted though that previous generations freely used the word black to denote something that was not very nice. As a child, I was once in a park with a younger child of African origin, when two elderly ladies came by. One said "Look, it's black". The other said "Yes, they've got them working in food shops now; it's disgusting". I found it as unpleasant then as I do now, but such attitudes were by no means unusual in those days. How much worse must they have been when the phrase originated?