Well if you both signed a paper saying you swapped then that basically states that you both accepted it as sold as seen sort of thing. He can try and take you to court but they will go in your favour as he had the chance to closely inspect the car. As for the phone calls etc just keep voicemails and txts for proof of harrasment and let him know that you are doing so. Hopefully this will keep him off your back but theres no telling with some people. At least if he knows you are keeping proof of everything it may make him think twice.