Forgive me querying your answer GetOverIt but is that strictly true about showing the time in France?
This is what wikipedia says and if this is true it does make the rhyme more sense:
Unusually for clocks in Britain, Little Ben is set one hour in advance of Greenwich Mean Time throughout the year. During the summer this coincides with British Summer Time, while in the winter this is the same as Central European Time. Although this means that the clock displays the local time for only half the year, it does demonstrate a preference for neither Britain nor Europe. This sentiment is captured in the rhyming couplet Apology for Summer Time signed J.W.R. affixed to the body of the clock:
My hands you may retard or may advance
my heart beats true for England as for France.