If you had asked before you even took this job on if it was a good idea I would have said NO.
Whenever you get a group of people like this who will say they will pay for something there is always a couple who cant or wont pay or who drop out, leavng the poor sucker who agreed to organize it out of pocket.
The problem you may have in trying to change the people is that the holiday is booked in their names and the holiday company may charge you for changing peoples names on the holiday details.
And YES, I do think you are being fair. If they are not willing to pay like everyone else throw them off the holiday. After all it is you who will be out of pocket.
Oh..and you have learnt your lesson, dont do anything like that again unless everyone pays up front.