There is absolutely no reason why you can't take your child on holiday ~ especially for only 4 nights.
He has the right to apply for residence, but will of course have to provide a reason why ~ taking the child on holiday will not be a good reason.
If he is worried about his private life being exposed, surely he wouldn't want to go to court again? if you are going to contest his claim I am sure the court welfare officer will do all the can to look into both family lives...not to mention the fact that your daughter has already expressed the wish to stay with you. He won't have a case.
Your ex IS a controlling bully by the sounds of it ~ call his bluff and take control of the situation, even if it means taking legal advice. Good luck :o)