Difficult to say, although if you can prove what payments you have made, it is unlikely. On the other hand, if your partner were to be the main carer for your son then that might make things different. You could try getting some advice from CAB, which is free, in the first instanc. But if you think your partner will turn nasty, you will need proper advice - you can usually get the first half-hour free from solicitors to enable you to decide what you will do next.
Good luck with it.