Assuming you are in the UK as you have parental responsibility you have a right to maintain direct and regular contact with your son as part of looking after his welfare, whether or not he lives with you.
Your partner who also has parental responsibility cannot prevent you from having contact with him unless she/he obtains a court order. She/he can oppose an application for contact or apply for an existing contact order to be discharged. The court will consider the best interests of your son when making a decision about stopping contact.
Even if the court makes an order for contact, there is no guarantee that the your partner will comply with the agreement or order. You could well find that arrangements are broken, excuses made, or your son fails to arrive. If your partner is determined not to allow contact, it can prove extremely difficult to enforce an arrangement or court order. If it is not possible to resolve the situation, you will need to ask the court to intervene.
I agree with clcandj and zgma but if you cannot afford a lawyer you should go to your local Citizens Advice Bureau who have access to many additional resources and whose services are free.