First thing I should say is do try all reasonable means to stay on good terms with your neighbours!
There should be little T marks on the property deed plan to say which fence is yours, but that only says which fence you own. Unless there is a covenant on the land to say you must keep it fenced, then all the T mark says is that you own the fence. You don't have to keep your land fenced (but you do have to control eg straying dogs if the dog is yours) so what you could do is remove the damaged panels entirely or leave them as they are if they don't pose a risk or aren't impinging in your neighbour's land. Go and take a look at the gardenlaw website for more info.