It (usually) refers to men. Assuming they are reasonably dark-haired and have shaved that morning, by the end of the day they have a 5-o'clock shadow - i.e. look like they need to shave again.
One of these was Famously credited with losing an Election for Richard Nixon. You could clearly see a 5 oclock shadow in one of his TV broadcasts. "A President that can't be bothered to shave, heaven forbid!"
There's also a '7 o'clock shadow'. That's seen on people who might shave at 6:00 a.m., yet by 7:00 a.m. it looks like they need to shave again.
I had a friend in the Army who had this, and he was always being queried on morning parade as to whether he'd shaved that day. He was only kept out of trouble by his corporal, who would truthfully declare to the inspecting officer that he'd seen my friend shaving that morning.