This is a very perplexing question. The fact that so many of today's top players did start out in their infancy (Nadal 3, Djokovic 4, Sharapova 4, Kournikova 5, Williams 4) would appear to support your discouragers, BUT it ain't necessarily so.
I would imagine a significant proportion of them would still reach the top (or certainly become professionals) had they not started until later.
I think you should go ahead, your enthusiasm will presumably be more than sufficient motivation for a few years at least and your progress at that time will help you decide whether to continue or not. You've got nothing to lose (except a few games LOL) so go for it - and if you don't make it, so what. You won't be the first to change careers later in life, and you will have enjoyed it at the time.
Persevere, ignore the "know-alls" (who are probably jealous of your strong sense of direction, and your determination and decisiveness), and good luck.