The answer to your question will depend upon what is meant by your phrase, 'It is our responsibility to replace it'. I assume that this means it's your fence but that doesn't automatically oblige you to replace it.
In the absence of specific regulations, there is no reason for you to have a fence between your neighbour's property and your own. You're perfectly entitled to leave the boundary un-fenced. If your neighbour was to complain, you'd have every right to say "If you want a fence, you put one up". Equally, you can put up a 3 foot fence, or any other size of your choosing - unless, of course, it was tall enough to require planning consent.
However, there might be a covenant in the deeds of the house which places an obligation upon you to maintain the boundary fence. If so, you'd have to consult the terms of the convenant to check the precise wording. (The height of the fence might be stated within such a covenant).
So, in terms of general law, your neighbours have no say in the height of the fence (unless it's so high that it's blocking out light) or even whether you erect a fence at all. If they try to tell you how high the fence should be, you should ask them which covenant they're referring to.
Chris