Hi again herons, A difficult decision!, if they have now returned, it looks like they are viewing it as a potential nesting site and in this case it would seem a shame to scare them off.
Maybe it's time now to forget your worries and just leave things to nature and take comfort in the fact that in the wild, without a bird box, they woud choose a hole in an old rotten tree which may have been used by generations of birds.
I must addmit, I sometimes forget to give mine a clean until it gets a bit late and they either take it or leave it.
One box I have in an apple tree for the last 4 years has only provided a home for a spider and otherones get used almost every year. Fascinating to watch them building their nests and then feeding their chicks! Good Luck Tbird+