My son was born in 1985 (now nrly 23 years). We split up in 1987. His father and I were not married, but he took me to court for custody. He was not allowed custody because we were not married (and he was a lousy father - proved in court - long story). He was granted access, on a saturday staying over till sunday. It lasted 3 mth then he got fed up and didnot get in touch for 5 years. When my son was 8 years old i got a letter to appear in court for access again. My son did not want to see this man he hardly knew but his biological father was granted access on a weekly basis again. He only had him 3 or 4 times and then left his life again. I was honest and open with my child and never put his father down and always made every effort for them to see each other - it was his father that never turned up for him. When my son was 18 years old he got into trouble and his biological father phoned him (after 10yrs) to play the loyal father - my son told him were to go and as not spoke to him since. He hates his biological father and adores his dad who has raised him since he has been 18 months old. BY THE WAY - his father left me.
I know all this does not relate to you, but i am just stating that is they let a bad father have access to his child in 1988, then i am sure that things have moved on since then. I am also saying that if he loves his children (as you say he does) then he must make every effort to keep in touch with them and you also. I wish you all the luck in the world and know that you will have a happy outcome.