Fair enough, but you may need to demonstrate what 'permission' you (and/or previous landowners ) have given this individual. Best if you have got a simple letter - nothing fancy or drawn up by a solicitor. Do you have such a piece of documentation?
Below is the reference document issued by the Land Registry that describes in lay-speak how such an application has to be made by someone claiming such a right. Obviously you are looking at it the other way around - ensuring that you take steps to prevent any further access.
http://www.landregist...des/practice-guide-52
If you haven't got enough documentation, in my opinion you'd be far better off killing this issue in TWO STEPS.
Step 1: Have a nice and polite conversation saying that you are happy for the access to continue right now, but you just want to put it onto a proper footing - then give him a letter outlining the access he is allowed.
Step 2: After a decent amount of time (say one year), tell him you now need to withdraw the consent - and do that in writing.
If, after you've invoked Step 1, he is minded to 'go legal' and claim a prescriptive easement, he will do it anyway now if you tell him you don't want him using the access anymore.
However, by doing as I suggest above, you will legally break one of the requirements for claiming a Prescriptive Easement, that the access without permission has been continuous - it cannot be continuous if you have invoked a break of a year putting it under a 'with permission' basis.
I cannot provide you with specific legal advice, so go to a solicitor if you are still unsure that to do. BM