I think hard as it is, you need to put what is right for your little girl first, not what you want. If she has been with your brother and his family for the first four years of her life, that's where she will feel she belongs. You might have given birth to her, but your brother is the one who has brought her up. You say she says she wants to live with you now - how can such a young child know what that really means? Are you sure you didn't encourage that thought? Surely you could have kept up your fortnightly visits without revealing your true relationship to her? It sounds as if you have been unsettling her and this is why your brother has stopped the contact.
Are you in a position to have her with you? Can you provide a stable and loving environment for her to grow up in? Could you honestly cope with a child when you have had no experience up till now of all that this entails?
Have you thought how your brother and his family will feel losing her after four years?
I'm pretty sure this won't be what you wanted to hear but you should think long and hard before you disrupt this childs life. I know children can be resilient but stability is so important. I'm speaking as someone who had none when I was a child and I'm still dealing with some of the issues arising from that.