I couldn't get any further into Cruciverbali's link above than to discover it refers to Brewer's Dictionary of Phrase and Fable. In that book itself, it says (quote), "The phrase probably comes from an old dice game" and later, "However it is traditionally held that...dispute...livery companies." So what should we go for...a probability or a traditional belief? It's almost a 'traditional belief' that the word 'posh' is an acronym of Port Out Starboard Home. It isn't.
The link in my earlier answer- from a noted etymologist and lexicographer - refers to the same matters, but dismisses the 'traditional belief'. So, effectively, does The Oxford English Dictionary and so, given the dates involved, should the rest of us.