Since January, I've had a weird growth on my tongue which goes and comes with varying size. It sounds a bit different to yours. At first they thought it was fungal, but the medicine didn't work and so I was referred to a specialist, who put me in for a biopsy. I don't smoke or drink either.
In my case it was a form of 'dysplasia' - essentially something going wrong in the maturation of the skin cells which can either gradually become worse (eventually tumor - but not without a fair bit of warning/progression if that makes sense) or (far more likely) it can go away on its own. Fortunately, mine appears to be doing the latter.
Bear in mind biopsies are pretty much only diagnostic - if your doctor was convinced it was something nasty, you'd be getting something else most likely. I really do empathise how uncomfortable and worrying it is to have a strange growth you can't explain, but that's pretty much the best way to find out what it is.
Be warned though - mine hurt like f*ck afterwards. I had to use some painkillers my dad had left over from his hernia operation. I'm not trying to scare you or anything but it's a precaution worth taking belive me!