Team GB is solely funded by the British Olympic Association and "the vast majority of the funds raised by the BOA are through the granting of sponsorship, licensing and other commercial rights that allow companies to become officially associated with the Olympic Movement in one form or another":
https://www.teamgb.com/legal/2gF98htIPgcfO3UgKxpthz
So there's no Government funding for the Olympics, per se.
Individual sports receive money from UK Sport to help their athletes train and to develop facilities for those sports but such funding isn't directly linked to the Olympics. Most of UK Sport's income comes from the National Lottery, with the bulk of the remainder coming from the Exchequer. However the amount of Government funding it receives is quite low compared with sports' funding in other major Olympic nations:
https://ibb.co/R6wNJBY
So money for the Olympics hasn't been diverted from public funds. It comes from companies and organisations who choose to get involved with the Olympic movement, with the athletes taking part having received some assistance with their training from the Government (and from the National Lottery), which they'd have received even if their sports hadn't been included in the Olympics.