To deal with your final point first, it will be many years (in excess of 20) before they could ever claim a right of access across your land.
It does seem you are being taken advantage of, and you must decide whether you want to stop it or merely regulate it.
If you want to stop it, you are going to have to just bite the bullet and tell them. To make that easier you could say that you are planning changes in your garden and you intend to fence up the access (wherther you actually do it or not). You should follow the conversation up with a simple letter 'confirming our conversation, no further access will be possible across my land from date x - perhaps giving them a month's notice if you want to be really kind.
If you want to regulate it, tell them by letter they can use the path for Y purposes only. That the permission is temporary but will remain in force until you confirm in writing that it is cancelled.
Of course some people prefer to get a solicitor to deal with this sort of mucky business by letter.