My son is a big strapping lad but is only 8 and I do not allow him to go to any public toilets alone (especially after the story here in the UK a few weeks back when a little girl was followed and attacked in a toilet in sainsbury's. ) I make him come in the ladies with us and as I have other younger children, it kinda looks ok, noone ever questions us or minds, and it is only him that moans, but frankly you're right; who knows what type of men go in the mens room? I don't envisage him going in by himself until some considerable time in the future, maybe 12 or 13...even then, I'll be worried sick!
Hope this helps.