I would say it is because your eyes are having to adjust from focusing in one place to another. So when the glasses are removed, the eyes are flailing about trying to focus on something!
Strabismus. Some people have cross eyes, or a lazy eyes, which is seomtimes corrected by wearing glasses. When the glasses are removed, the eyes gets confused and reverts to its old habit.
As a glasses wearer I can tell you that first of all it's the shock of the difference between clarity and fuzziness. and then secondly it's trying to focus (and failing miserable). Rather like when ppl try on other ppl's glasses and end up looking incredibly startled and blinky!