We had a cat many years ago and she often came back without her collar. We thought someone was stealing them until a cat-loving neighbour we knew assured us that cats often remove their own collars. Many cats don't like being encumbered by a collar as it affects their ability to assess gaps to squeeze through. They also feel it's a hinderance when climbing or jumping. According to this neighbour, many cats are adept at removing collars by snagging them on something and then wriggling out of them. Most owners don't fit the collar tightly as they feel it might be uncomfortable. However, that gives the cat enough of a gap to squeeze out once it's snagged the collar on a branch or something.
Hope that helps Tigwig. Your cat's collar isn't being stolen by the local kids as we thought, he just doesn't like it and he gets rid of it when you aren't about.