A certain amount of cold weather is good for olive trees!
Quote 1:
"In order to initiate flowers and fruit, olive trees need a two-month period of cold weather (with temperatures below 10°C (50°F). They also need a fluctuation between day and night time temperatures. Plants kept indoors are therefore unlikely to flower".
Further, a mild frost is unlikely to do them any harm.
Quote 2:
"Olives are not entirely hardy in the UK, and will be damaged by temperatures below -10C (14F)".
Source (the Royal Horticultural Society, who ought to know what they're talking about!):
http://apps.rhs.org.u.../profile.aspx?pid=138
Chris