No it's down the individual property - we owned a house that had no responsibility for any of it's boundaries - just the front wall - even though it was boardered on the three others by fences and walls. Sometimes you can have an incline by the "good" side of the panel and which way it faces but these are only guidlines.
Even if it's your neighbour's that doesn't necessarily mean they must replace it. They may own it, but they could choose not to have a fence at all. "Whoever wants it the most" is about right.