Assuming the fence is yours (according to your deeds, and you maintain it as necessary):
Tell him you like the ivy and you intend to keep it, but if he wants to trim back any that grows over the boundary then you have no objections. He has no right to tell you how tall it should be, but he does have the right to trim back any that overhangs into his property.
He knew the ivy was there when he planned the conservatory and should not have assumed that you would be willing to get rid.
If the fence is his, that's a different story and he can insist that you remove it from his fence competely.