Er, you don't live a couple of doors along the road from me, do you?
The moggy from that house has discovered that my front door has got a cat flap (whereas his own front door hasn't). Further, he's worked out that a simple miaow from him will get me to open yet another tin of cat food. So his official 'owners' probably don't see him for days on end. (He seems to spend most of his time asleep on my settee, asleep on top of my wardrobe or - like this afternoon - knocking all of the ornaments off my dresser).
However if I'm way from my house for any length of time (so there's no one to feed him), he'll then remember where his other home is, and return there for a brief period.
Your feline friend is simply getting fed somewhere else but returning to you when the alternative supply of food isn't available. The only way to get him back (other than locking him in) is to install a cat flap - if you've not already got one - and to ensure that he has constant access to his favourite food (and, of course, to fresh water).
Dogs are loyal to people. Cats are only loyal to their stomachs ;-)
Chris