If the fan feeds into a length of ducting that takes the stale air somewhere to vent, it is very possible that the warm and moist air from the bathroom is being condensed in the ducting, the water (complete with brown non-bat gunk from previous extraction) is dripping back down the ducting into the fan and out.
It's the cold temperatures doing it - solve by insulating the outside of the ducting with fibreglass, if you can access it.
(Clearly this possible explanation doesn't apply if you are extracting straight out the window)