If a cars only done 4k/year then i'd guess a lot of this will be popping to the shops, ie cold engine, lots of gear changes etc etc, so the engine may well be in a worse condition than the one thats doen 8k/year and had more runs at 'operating' temperature, not always the case obviously. Personally if they're the same car 1'd go for the newer one everytime, remembering the newer one could well have had more improvements on it from the manufacturer ( i read once ever year manufacturer's change lots of parts on the same model of car)
Also as you say the '02 on's sellers sounded dodgy then i'd definitly go for the 04 model.