My advice is that you are under a lot of stress at themoment especially with your father being terminally ill. If you feel that your son is safe with his father, then let him take care of him as often as he likes this will in turn let you spend time with your father. If at a later date you do go to court how can your ex say that you have not let him see his son. Men play mind games, so play him at this own mind game, let him think he is winning, be nice and smiley and this will throw him off course and also take pressure off you. He probably wants to come back after finding out the grass is not always greener. You are still hurting and your wounds raw. Take time out to heal.
Been there, done that, got the t shirt. My daughter is 28 and the last time she saw her dad she was 7. he played the same game and as soon as I let him have his own way he dropped us both like hot potatoes. I am now married to a wonderful man who has been twice the father her blood father has been.