I can't find an absolute derivation, but it certainly goes back to the 1740s. I suspect the answer may be that when speaking to a pedestrian from horseback one assumes a commanding position ... and anyone on a horse was likely to be rich and ill-mannered. Sorry not to be of more help.