You dont say what type of holiday. Is it driving in a car to somewhere in the UK, or flying in a plane to somewhere abroad.
Anyone can drop out of a holiday, but I guess this may upset your other family members. They may be loking forward to a famly holiday.
They may also worry about you back home while you are way and it may spoil their holiday.
How old are you, and what is the problem with going away with your family?.
I can remember when I was about 16 or 17 I just did not want to go away with my family, and I would rather stay at home than go away with them to italy.
But I must say you have left it a bit late.
Instead of telling them you dont want to go, why not talk to one or other of your parents (or another family member) and just say you have some concerns about going on the holiday.
If you just say you dont want to go it may start an argument, but if you just say you have concerns hopefully it can be discussed in an open manner rather than becoming a family argument.