The professional advice is the first time he gets out of bed say 'It's bedtime darling, go to sleep' as you're putting him in bed, the second time he gets out it's 'it's bedtime', then the third, you're supposed to say nothing as you put him to bed.
With my son I made that mistake too of letting him sleep in my bed, the way I got him out of it was by sitting at his bed the first time he falled asleep, then in the middle of his bed the next night, it felt like an eternity til I could sit at the bottom of the stairs but it worked. He just needs to know that you're there. My son's 7 now and has been sleeping (almost blissfully, alone) for the last four years.
It just takes time and although it does hurt as a mother to hear them cry, your partner is right, they're not hurt or being emotionally scarred, they're just trying to get their own way.
Kate x
P.S. You health visitor suggesting that your partner deal with him is ridiculous, your son needs to know that you're both doing the same thing, so maybe one taking the reigns one night, and the other doing the next night would be best.