Yes. Note 6 in the current passport begins " This passport remains the property of Her Majesty's Government in the United Kingdom and may be withdrawn at any time". I expect that the commonest reason is that it has been obtained by some deception or misstatement but the Government plainly retains an overall power to withdraw its own property, the passport, from the holder and warns us so.
Aptly, so called 'football fans' can have their passport taken off them. Maybe only for the duration of the tournament. Others can be deported. The complete answer from our FredPuli.