I don't think, reading this a bit more closely, that the colour per se is the whole problem.
It's probably a rather superficial way of looking at things in fact. Many of the "asylum houses" are the only non-derelict houses in a terrace. Plus the fact that the people probably stick out like a sore thumb anyway.
But if you're the local indigenous bored thicko, maybe red doors trigger some primeval instinct ...