Your mortgage company has to be satisfied that your can afford the mortgage on your salary alone - they don't care that you have been paying it on your own anyway, if the figures don't add up they won't agree for the second party to be removed.
As it stands at the moment, they have 2 people to chase for money if the mortgage isn't paid, if they remove your ex then they have only one - you.
Have you got a job? Does your ex pay maintenance for the children? Don't wish to be nosy but how much is the mortgage and how much do you earn?
You may have to sell up and rent.