Being a bloke is hard enough without feeling you are competing with someone your girlfriend went out with for years. Who probably had a bigger ahem, was romantic, didn't scratch his crack at the dinner table or burp the alphabet in one breath, got on with your friends and didn't make passes at them (or your sister even though this should be allowed) even though they probably all fancied him. Oh, and who was the best lover. Like ever.
I think you'll find that you'll always naturally compare current with exes - some things about him might never be as good but some will be better. If that's not the case, he probably isn't right, especially if you are articulating negative comparisons about him. Andy's response is put the most succinctly.
Whatever you do, don't tell the new guy what your exes used to do/say/wear etc. or what "your song" was but that should go without saying.