My bloke had it a couple of years ago, for the second time. He's only 55, but is classed as 'at risk' by our GP because he suffered from pneumonia a couple of years ago and has been left with recurrent asthma. Previous to that, he never caught 'just a cold' but always an infection that manifested in his chest and laid him low for at least a fortnight. Since having the jab, he's suffered no ill effects other than a sore arm for a few hours, and he certainly hasn't been brought down with the crippling colds and flu he's suffered in the past.
In the end, it remains your decision, but do talk it over with your GP/practice nurse before deciding.