First check what your deeds / Land Registry entry say. If these reflect, your belief and what has been the practice for 46 years he cannot change them.
If however the deeds and actual practice differ, you can claim adverse possession, ie you have used the land for 46 years without challenge so you have become entitled to do so by custom and practice. You can challenge your neighbour by quoting this to him but may need to see a specialist solicitor to sort it out. The solicitor should also be able to amend your deeds.
If this applies to all the terrace you should be able to club together to use one solicitor to save money.