I always took the attitude, work is for earning the rent money, friends are people I choose to be with. I will be friendly to people that I work with, but they aren't friends, and I don't seek them out after work hours. So first of all, don't worry unduly about these people, just do your job.
Many workplaces develop a thing about groups going out socially. In my opinion these become toxic, with relationship issues and 'who said what' spilling over into the workplace. At the same time, these groups become a way used by certain individuals to bully and exclude someone. I'm wondering if you're receiving a first dose of this.
You say you were upset about this, and that's important. Without knowing anything about the place you work in, what to do about being made to feel upset is hard to advise on. Is there a supervisor, manager or colleague who isn't involved closely with this group, and who you trust, that you could talk to in the first instance?
Just expressing that you aren't unhappy, that you are a quiet socialiser, and that you feel a bit hurt is something that might help all round.