Have any of your friends asked him what his problem is with you?? They seem keen enough to tell you what he's been up to, and what he's said. It seems strange that if they're so keen to let you know all these terrible things he gets up to, they haven't established why.
Playing a little devil's advocate here but - are you sure that he does all these terrible things? or could it be that the villagers have take a dislike to your neighbour, and not him to you? Just a thought.
I hope you manage to sort something out - I would try to be straight with him, call round one day and ask him around for a drink in the garden or something - make out you know nothing about what's been said by the villagers, and make your own mind up - and let him get to know that you're not the terrible person he seems to think you are.
Good Luck ! x