I know this is weird but use a scrubber-the ones you use in the bath for goodness sake!! Not under ANY circumstances a wire one!!
No...I mean the light brown coral kinda ones...loofah I've just been reminded!
I have a seperate one for my face as I find them fantastic for just getting the dead skin off etc, I don't get as many spots actually either. Just wash your face after you've been in the bath etc, as you do it, just use the loofah, but not too hard. Just enough to surface and cleanse fully.
Don't go rubbing your eyes with them either...oopsies!