It started life as 'chewing the rag', which was a piece of military slang meaning to whinge/complain rather than just to chat. It first appeared in the 'fat' form in an article in 'Punch' magazine in 1916. The whole idea of the chewing element is in comparison with a dog 'worrying' endlessly at something.
Or alternatively, the explanation I read recently (without references to original transcripts I must say) about life in the 1500's in Britain:
'Sometimes they could obtain pork, which made them feel quite special.When visitors came over, they would hang up their bacon to show off. It was a sign of wealth that a man "could bring home the bacon." They would cut off a little to share with guests and would all sit around and "chew the fat."