Unless there is a covenant in the deeds about keeping the fence in a good state of repair, nobody is ever obliged to maintain a garden fence. If the fence is their property they're free to let it fall down or to simply remove it altogether. You can put up your own fence, either immediately inside the existing one or, with their permission, by replacing your neighbours' fence. It will then be up to you to decide whether to maintain that fence or simply to let it fall down.
The foregoing assumes that you've got the ownership of the fence correctly established anyway. It's a MYTH that you always own the fence which is either on the left or the right as you look out into your garden. It's also a MYTH that the house owners with the fence posts on their side of the fence always own that fence (although it's often a good guideline, since people who erect fences around their gardens usually put the fence posts on the 'inner' side). Check the deeds to see who owns the fence and who (if anyone) is responsible for maintaining it. (It's not always the owner who is responsible for maintenance. There's a covenant in the deeds of my house which states that I'm responsible for mowing the lawn around the corner, even though it's not part of my property).
Perhaps you could ask your neighbours to move next to me? We'd get on well (if we could ever see each other through the undergrowth!) as I've barely touched my garden for the past 20 years. I simply don't see the point.
Chris