YM. If I found out like you did that my ex had left our son, aged 3, alone, it would be a drop dead deal breaker. He would never see his son alone again. No 'but he's his Dad!' Your son's interest is no1. The best predictor of future behaviour is past behaviour. If you leave him with him again, he WILL leave him on his own. He will also tell your son 'things about you' as soon as he can also, as he is a spiteful moron who cares only about himself and not a jot about the welfare of his son. So don't let this intimidating threat hang over you either...it's a form of blackmail and you can't let that happen.
Why were you ever with him?????????????