I think you should make it clear between yourselves what is going to happen over the days he has them when you will be dropping them of and when you are picking them up, you need to agree on arrangements that suit you both.If he wont agree to something which makes you both happy then stay put in your home until he can be a bit more grown up about the situation, i think the last thing you should do is let him have the children before coming to some agreement on the arrangements. If I have understood you correctly it seems like he is using the children once he as them in his care to get his own way, this could partly be a power struggle against over you which really there is no need for.