The odd 20p difference now and then is probably just due to 'rounding errors' in the system. For example, I stated that you're allowed £124.52 per week of tax-free pay. It's possible that some pay systems might actually allocate £124 of tax free pay to one week, and £125 to the next week, on an alternating basis (with either the first or last week of the year being used to 'tidy up the last few pennies'). If so (during periods when you're working exactly the same number of hours each week), your tax will vary by 20p each week.
Employees who have a regular working week (with the same pay each week) should normally expect to pay the same amount of tax each week (irrespective of whether they're at work or on holiday). But you seem to be working on the system of 'stored holiday pay' which is often used where employees work irregular hours. As you've found, that can result in you paying more tax in those weeks when you've claimed you're holiday pay. (It's not unfair. At the end of the year you'll still have paid exactly the same amount of tax as a 'regular' employee with the same total earnings').
To enable me to explain, let's consider the very first week of the tax year. I'll assume that you worked for 36 hours at £6 per hour. That means that you've earned £216, of which £124.52 is tax-free. So you'll be taxed (at 20%) on the remaining £91.48, resulting in £18.30 being deducted from your pay. However, in order to comply with the legislation on holiday pay, explained here
http://www.direct.gov...dholidays/DG_10034642
your employer will add 12.07% of your gross earnings to your banked holiday pay. That means that £26.07 will be 'banked' for you. But that money is ALL in excess of your tax-free a