You put your left, weaker, foot in the stirrup and push up with your stronger, right foot, hence you stand on the horse's left. Someone will say that they push up with the leg that is in the stirrup, but when people rode with longer stirrups or used mounting blocks, that wasn't so easy. Can't say as I've noticed horses, whatever their temperament, being more twitchy when approached from one side rather than the other but Chris's link seems to suggest otherwise.
All this is one reason to drive on the left. The world over, people mount horses from the left , which means the horse at the kerbside was already on the left of the road and would move forward 'driving' on the left.