It can be construed as being patronising, even if it is not intended to be so.
The fact is, as a polite society, we should be mindful less of our intentions in using speech patterns, and more of the way they are received. I have always desteted being referred to as 'young man', I still get it now occasionally, and I am a grandad of 53! I also dislike being called 'mate' by strangers, it is a matter of receipt rather than transmission, so I would never refer to a woman as 'love' - it's not a phrase I use.