'Gaff ' was originally a colloquial term for a fair (recorded 1753) thence it became the word for any place of public entertainment (1812) and then the cheap music hall or theatre sometimes called a penny gaff.With a wry sense of humour, Londoners started to compare a pub with a cheap entertainment upstairs with any particular building and specifically a house.[recorded from 1932]
Curiously, Londoners who still use 'gaff' always seem to say say 'the gaff' not ' a gaff'.Can't think I've ever heard it used otherwise than when referring to an address or building which had been mentioned already or the identity of which was already understood by the hearer. Burglars don't say 'I went out and blew [that would be a very old burglar!] / did a gaff' but will say 'Y'know that red warehouse, well, Tommy only went and did the gaff!