A bridleway is not part of the public highway. It is thus a criminal offence under S34 of the Road Traffic Act 1988 to drive a motor vehicle over it. It is possible that you find that owners of property further up the bridleway seem to be driving motor vehicles over that, but they are able to do this by claiming a prescriptive easement. To do that, owner or his predecessors must be driving over the land unchallenged for at least 20 years. Since the owner of the bridleway has identified you doing it and challenged you, there is little you can do - his is right, unfortunately.