I've got news for you.
IT'S NOT JUST MEN!
I've seen women in both flat sandals, AND high-heeled sandals, with socks. And to be honest, I'm not crazy about the tights-and-sandals combo on women either. But no-one ever comments on that, as though it's in some way acceptable.
But yes, whichever way you look at it, socks and sandals is not good. I don't even like seeing people in shorts, trainers and regular socks. A neighbour of mine recently went out in shorts, black socks and (office) work shoes! I'm sure I did a horrified double-take.