I wonder how long you have been there? - within one month you are due a written statement of particulars of employment. This does form part of your contract - although it won't detail enough of your present query to help you - it will (at least it will if done properly) include details of how much holiday you are entitled to. This is the rest of what it should include:
http://www.berr.gov.uk/employment/employment-l egislation/employment-guidance/page16367.html# What%20information%20must%20the%20written%20st atement%20include?
Contrary to what you currently believe, do DO have a contract - its a combination of statutory employment law (which the employer must follow), oral contract (things said to you) and custom & practice (what the employer has being doing before). See here for some confirmation of what I am saying (that you might want to print out and quote at your (apparently useless) employer.
http://www.berr.gov.uk/employment/employment-l egislation/employment-contracts/index.html
So if your employer did this last set of bank holidays and/or has been doing it a while, there's probably not a lot you can do. If this is the first year, it was sprung on everyone at short notice, he's probably in breach of his contract.
What power you have to do anything about it depends - on you, plus how long you have been there. How long have you been there? - outline that and I'll outline some options. BM