Elsie i'm sorry about your situation he sounds like a nasty evil piece of work, my friend has a 5yr old boy and her situation was pretty much the same as yours apart from the older daughter part, her x was seeing 2 other women kicked her out of her house when she was pregnant although it was after months of physical and mental abuse he even threw her down the stairs when she was pregnant threw things at her e.g chairs, knives
when she had their little boy she moved away changed her name as she was scared for her and her little boy, anyway he found her and they went to court, he now has weekend overstay access and the reason they gave my friend for actually giving him access is he's never abused his child
i do hope if your x goes to court they do not give him access, if i was you i'd find a very good solicitor