If anyone can work out cat psychology then they deserve a medal :)
One of our cats is like that, our girl cat is the most loving, soppiest thing ever yet the other, a boy cat, most people are too wary of his ability to suddenly turn and get nasty to risk lavishing too much affection on him though he is getting better as he gets older.
He will let my brother roll him over and rub his tummy for ages whereas anyone else who dared try beware lol.
It may be a protective/attention/territorial thing, he is attached to you and feel threatened by the attention, our female is like a guard cat when it comes to mum, puts herself between her and any stranger. If she wants a lap etc... it's always mum.
Did you rescue them? Could be something connected to their past in that case if they've had a bad experience with males.
If he wants to try and get round the cat them he could try carrying some catnip round in his pocket when round the cat, sitting with an open lap if it's a lapcat or putting a blanket or similar by him when he's sat watching the TV etc...
Advise him not to stare at the cat as they don't like that, blinking at a cat can help, just an obvious slow blink then look away.
He may be giving off signals back to the cat so he could consider the way he talks to it, if he does, tone of voice etc... and body language towards the cat.