Does he not realise it is actually ILLEGAL to leave a child of that age alone?
I hate to say this, but I think you, and your children, could actually be better off without him around.
You are going to do nothing but worry whenever he has the children (Im including the one on the way too) and whenever you need to "discuss" something regarding the children with him, he is always going to use emotional blackmail against you.....ie...start a new family elsewhere.
Please dont hate me for saying this, but he doesnt deserve ANY family, especially you and yours. Not only has he put his son at risk, and who knows how many times this has happened, but he is also causing you unecessary worry and stress that no one needs, pregnant or not!
You almost seem scared of him.....has he ever been violent to you in the past?
As hard as it seems, you need to put you and your children first. I sincerely hope he can be a part of your family, but not at the risk of your children.
Good luck, hope it all works out for the best.