You grass will be registered at the Land Registry - it has to be because of recent rules forcing land to be registered on first transfer. But can you confirm that the farmer's land on the other side of the boundary is registered or not - this could end up being important. Post back if you don't know how to find this out for a few £s at LR and we will tell you.
Secondly, have you got a copy of your title plan - this is the plan that LR uses to register the extend of your land - it is fringed in red. Get it from the LR if you don't have it yet.
Thirdly, you need to know that LR treat their marked boundaries in red as 'General Boundaries'. This means they are not definitive, but an approximate position. These days LR maps from the air against the OS maps and the boundaries are marked accurately to within about 0.5m - though LR will never admit their accuracy. It is possible to go through a process called 'Determining the boundary' using the LR. It costs, but it may be a solution for you.
Alternatively you may have to engage a solicitor - if you are not comfortable with finding out the things that I have suggested doing with the LR.
Your landholding at 4 acres is about 16000 square metres, so we are talking about 1.5% shrinkage - you need to think whether it is worth investing a solicitor's time in fighting this - unless you are comfortably pursuing it yourself.