The fact that you admitted to pushing the other guy does not, in itself, amount to an admission of committing an offence.
If you said, "I couldn't listen to any more of his insults, so I pushed him", that might well be an admission of common assault.
However, if you said, "I thought he was going to attack me, so I pushed him away" that would simply be a statement to the efffect that you had exercised your legal right to protect yourself.
Anyone who believes that they're about to be assaulted has the right to defend themself. The belief does not have to be 'well-founded'. (i.e. it doesn't make any difference whether or not the other person was actually planning to attack you. The fact that you thought it likely provides sufficient grounds for acting in self defence).
There are, of course, conditions which apply to the right to self defence in these circumstances. First of all, you have to believe that the threat is immediate. Secondly, your actions have to be 'proportionate' to the perceived risk. From your description of the incident, both of these conditions were met.
You should seek a solicitor's advice but, as I read it, you've not committed any offence.
Returning to your original question, if you refuse to accept the caution, of course the police can charge you. Whether or not they would do so is unclear. However, unless you made a completely self-incriminating statement (under caution) to the police officer, I doubt that they'd want to take the case any further.
Chris