I agree with Reverandfunk, you don't have any reason to be in contact with him, and to be honest, I don't think your partner is being unreasonable to ask you to not talk to him.
He might have self-esteem issues, and be worried that your ex was better looking or a better lover or whatever. And is worried that you're gonna run straight back into his arms.
Why did you break up with your ex in the first place may I ask?
You also need to talk to him calmly, tell him that you love him and no-one else, and you're not gonna walk away. Try not to get angry and shout it out, makes it less believable doesn't it?