Family courts do tend to give mother more status than fathers with custody battles, however contact is a different issue. It is recognised how important it is for a child to maintain a relationship with their father and it is considered the right of the child (rather than the right of the parent) to have this opportunity.
If, as you say, you have bended over backwards and consistently attended contact, travelled to him and so on - then this proves to the court that you are a committed parent to your child. Get a good solicitor and you should be granted what you are asking for as it seems very reasonable!!
I wish you the best of luck!!!