The amount of data used to watch Youtube videos depend upon their resolution. (i.e. whether they're 'standard definition' or 'high definition').
As a very rough guide though, watching an hour of SD video uses up about 1Gb of data, while watching an hour of HD video uses up 2Gb.
Remember that anything else you've got connected to the internet (such as a NOW TV box, or similar) will also eat into your data allowance.
I'm amazed that BT, which actively promotes services based upon data streaming, even offers a 'capped' broadband service these days. Many other ISPs did away with caps ages ago and now only offer uncapped services.