Here's one indepth explanation from World Wide Words:
It�s originally British.
There are actually two common idioms based around the phrase a kettle of fish. One, means �This is a different matter from the one previously mentioned�. The other is more of an exclamation: either as a pretty kettle of fish! or a fine kettle of fish!, meaning that some awkward state of affairs has arisen. The latter is much older, dating from the eighteenth century, while yours is twentieth-century and seems to be derived from it.
Nobody is really sure where the expression comes from, but we do know that the phrase a kettle of fish was originally a literal term. These days, especially in Britain and Commonwealth countries, we think of a kettle as a small enclosed container with a handle and spout for boiling water to make our tea. (I believe that Americans are less familiar with this essential item of kitchenware.) In the eighteenth century, though, a kettle was any large vessel used to boil stuff in.
There was, it seems, a custom by which the gentry on the Scottish border with England would hold a picnic (though that term was not then known) by a river. The custom was described by Thomas Newte in his Tour of England and Scotland in 1785: �It is customary for the gentlemen who live near the Tweed to entertain their neighbours and friends with a Fete Champetre, which they call giving �a kettle of fish�. Tents or marquees are pitched near the flowery banks of the river ... a fire is kindled, and live salmon thrown into boiling kettles�.
Contd.