I think someone once told me that it stood for Port Outward Starboard Home, like the sailing/shipping terms for left and right (or is that right and left...hmmm) but I'm not entirely sure how that links to fancy folk! Sorry!
It is definitely not from "Port Out Starboard Home" - that is a myth which has been thoroughly investigated by George Chowdury-Best of the Oxford English Dictionary, and who found no evidence for it in the form of documents, tickets, contemporary references in newspapers or letters etc.
It is probably from late 19th century posh = dandy, or from Romany posh = small coin
The Romani word POSH means Half, not coin. However POSHES is sometimes used for money, but it is far less common that the word Wongar (wonga).
Personally and despite Chowdury best, I think the evidence for the 'port out starboarc home' trips that have been proved to have been popular is the most likely source.