Is it that you don't want people stalking you or other wise bothering you? Cos if so, you need to say this to your line manager - it's a reasonable fear. One place i worked at allowed front of house staff to have badges with 'work names' on that weren't their real names, so public couldn't take the 'I know where you live' line with them, and it might be that yours does similar. I am afraid however that being badged up is just an expectation now. In a sense, it can also protect you - it says you are officially on business for XYZ and not some random nutter. Increasingly these badges also act as magnetic door passes and so feed into staff time monitoring, and it really is no good fighting it if your works is going down that route.
Plus, I think your point about an ID number is reasonable - put it to your manager reasonably, or ask your union rep to do it for you.