I feel for you Ol. My ex threatened to abduct my son, who isn't actually biologically his but my other child is. That also got reported to the police, the only thing he got had for was the criminal damage he caused at the time. I continued to allow him to see the kids, eventually unsupervised but with lots of check ins.
He is emigrating to America, if he thinks my kids are going over there with him, at the ages of 6 and 2 for a holiday, he has another thing coming. Any holiday that doesn't involve his mum being there with him, is just plain out of the question.
Your toddler is only two years old, I think a holiday away from mummy is a daunting prospect. Even when my ex and I lived together, me away for one weekend was too much for my then 4 year old.
I will also add that my ex is not normal ex material, he is a bit of a violent lunatic when it comes to me but we do not know this womans full circumstances. I doubt a single one of you would reprimand me if I made the choice to prevent my ex seeing the kids, and she could be in similar circumstances.
Equal parental responsibility does not automatically give you an equal share of time. I think you need to get a formal agreement regarding access.
Baby needs to get used to staying with daddy, a weekend to begin and a build up until a holiday can be taken and you can prevent your ex leaving the country with him very easily until you are comfortable with the arrangements.