We work flexi time and have to be in between 8am and 10am - I came in at 9.10am which was when I took the temperature reading, but there were others in my room from shortly after 8.
The legislation says "within one hour of work starting". So, if your employers want to employ people from 3am, they have to make sure the building is at the required temperature by 4am. The fact that others may start later does not alter this.