I assume that your aerial feeds into the EE box which, in turn, provides a signal to the TV (probably via an HDMI socket), with you using the remote control for the box (rather than the one for the telly) to change channels. If so, the delay is produced in your EE box.
On the other hand, if you've got things set up so that you use the remote control for the telly to change channels, then the delay is produced in your TV set.
In either case the, actual cause of the delay is most likely to be a weak(ish) signal coming through your aerial, meaning that it takes a while before the relevant device can fully 'lock onto it' and display a picture. (Many TV transmitters have had their power output reduced in order not to cause interference to 5G mobile phone signals. Additionally, the signal you get at your aerial is affected by atmospheric pressure, precipitation, path obstructions, the 11-year sunspot, individual flares on the Sun's surface, local interference and much more). A cheap signal booster might improve things.