I read into it that the group itself are close (cliquey??) and you are not really keen to become part of it. If so I guess you feel bad that you went and said yes when you shoulda said no....doh!!!
Is it this weekend? You need to be a bit 'last minute' at this stage as you have already said yes but perhaps you can ring tomorro and say you need to drive your brother to Leeds (??), you owe him one and you feel silly but you actually promised him weeks ago. Keep it to-the-point and don't let them ask any more questions!! Just put yer foot down. Good luck