I was shocked when I read your post, I believe you should stop him seeing them, to hit a child accross the face is inexcusable, the possibility of brain damage comes to mind, apart from the psychologial damage.
And as for smacking her bare backside, she's too old for that, why not ask her if she likes going to see him, if she wants to, threaten him with the law if he carries on doing it.
He doesn't sound like a loving father, more a bully and pervert,
Sorry for the strong language, but thats how he comes accross.