(Assuming that you're in the UK) you're free to move anywhere in the country and your ex-partner can do nothing about it, irrespective of whether he's got parental responsibility or not.
If your ex-partner doesn't have parental responsibility, you can also take your children out of the country without his permission.
If your ex-partner has parental responsibility and you don't have a residence order you can't take your children out of the country (even for a day) without the permission of your ex-partner.
If your partner has parental responsibility and you've got a residence order you can take your children out of the country for up to a month at a time without his permission, but you need his permission for longer visits abroad (or to emigrate).
Your ex-partner will have parental responsibility if his name appears on your children's birth certificates (but only if they were born on or after 1st December 2003 - that date applies to births in England and Wales; there are different qualifying dates for Scotland and Northern Ireland). He can also have parental responsibility if it's been granted by a court (or if you've entered into a formal parental responsibility agreement with him).
Chris