Yeah, whatever happens don't give up on your daughter. At some point you may think she'd be better if you didn't put her through this upset, but in the long run that would be the worst thing. You're her dad and she'll never have another. I bet the problem is that you only get to see her for such a short time, and that the visits are so far apart. She might be thinking of you a bit like a stranger and would obviously be more comfortable with her mother as she's with her all the time.
Are you not allowed to see her more often? Would her mother not be willing for you to see her more - to try to make it easier and not so upsetting for the daughter she loves too?
I don't know all the story from both sides, I know but this little girls needs are the most important thing, and both you and your wife should keep this in mind when sorting this out. Try to put any ill feeling if there is any, aside for her sake