well if it goes through court which is what it sounds like its a whole other ball game.He has to see him often.Because your son is only three,your son can't choose for himself wether or not he wants to see his father.
i don't mean to sound deppressing but he sound like he is going to make things hard for you.
It would be fantastic if you could sort it out between you an not in court.It is a real long and destressing process.I should know I am in the process still after almost two years.
I would just go personally.But please get advice from a solicitor.It won't hurt to double check.I am sure you have no problem as you will need support from your family