of course you could have refused him entry, pretty much the only people you have to give access to your house is the police with a warrant!
when you signed up you probably agreed to allow them access but it also probably said with reasonable notice or similar, are you sure you've not missed a letter telling you a date they would be calling?
if you felt threatened then complain to your supplier that you felt intimidated by the meter reader and he effectively forced his way into your property