I know a woman who wanted to split up and take the children 100 miles away. The courts agreed they had no power to stop her, even though she had been married and was in the middle of a divorce. Unless the parent with custody is being completely unreasonable or irresponsible in wanting to go ( say ) to a very distant country, the courts will not interfere. They might be very interested in the fact he does not help maintain the child, though. He is unlikely to go to court, because if he did the courts would almost certainly order him to pay maintenance - even a token amount if he is out of work.
Go, and start your new life as soon as you can, and good luck to you.