Keyplus is right; gora/gori does mean white man/woman.
However, as a white woman who had a 6 year relationship with a Hindu man I can tell you we were both subjected to racist abuse from all kinds of people and it can be a term that is sometimes used in a racist context. I know, I was called it enough by Asians, (along with the derogatory, sneering looks) who didn't think I understood what they were saying.
It isn't necessarily a racist term though and a 1yr old child does not understand what prejudice is.
That said, I abhor racism and I get really angry when adults pass their prejudices on to innocent children. I will not allow my young daughter to be influenced by people's racial prejudices and they soon get put in their place if they are racist in front of her.