There may possibly be some connection with the fact that a glove was involved in a challenge to a duel in times past. Thus, we have phrases such as 'throw down the gauntlet' = make the challenge and 'pick up the gauntlet' = accept the challenge. So, the to-ing and fro-ing of gloves had an element of animosity involved.