there was a time when the clearly mentally retarded where considered more likely to commit crimes (i'm talking mediaeval etc) and were always blamed.
there was a fear that the devil had made them that way and therefore they must have an inherrent evil.
perhaps even now there is some underlying subconscious sense of this - that they perhaps haven't the mental capacity to tell right from wrong or are easily lead into a life of crime.
even now it is sometimes easy to tell when someone has some sort of mental disabilty just by their face.
in terms of disfugurement by injury rather than birth, i think this is a cliche -'' to be facially scarred you must have been up to no good ...''
and as has been mentioned its a hollywood favourite.
also, we like to think we can tell what a person is like by reading their face, their expressions etc - if something is preventing this, and making those expressions hard to read, perhaps we get an underlying sense of mistrust, however irrational and subconscious
it is an interesting subject