Kep chop or saw though the stems at ground level. This'll let it die off. then pull it away, once dead, from whatever it's clinging to. It may damage the brickwork/mortar if you're too eager. The bits that are left will dry out an can be brushed off with a stiff broom. Good luck!
Another solution, is to buy from a Garden Centre a gel thats called 'Ivy Killer', you paint it on the leaves, the poison goes down and kills the roots. Its good, i've used it, and the Ivy never returned, this is now two years being Ivy free.