Is there any reason why your cat keeps visiting this couple? There may be one or more things about your home which your cat has started to dislike, whereas it finds the other place more congenial.
1. Do you continually play loud music? Cats like peace and quiet.
2. Do you have noisy children? Again, these often make cats nervous and jumpy.
3. Does anybody in your house smoke? Cats often do not like the smell of tobacco.
4. Do you work which causes your cat to be shut out all day, in the rain which it may not like?
5. Is your cat given plenty of attention or neglected? It sounds as if the couple who are encouraging him may be retired and have plenty of time to fuss him, let him sleep in the dry undisturbed wherever he likes and let him be the centre of attention whereas perhaps he might not be in your house.
Cats are very fussy creatures. Sadly, you can never really "own" them. They have minds of their own and if they feel somebody else is treating them better, they may well desert. I hope the police can help sort this issue out for you, but short of keeping your cat locked in all day, which would be very cruel, especially in good weather, I'm not sure how else you can resolve the problem,