While I might not have been as advanced as your grandson, I loved numbers (and everything to do with them) from a verey early age.
At around 8 years I would get my Dad to carry a notebook when we went into town, on the top deck of the bus, each Saturday morning. I'd point out each car we were passing and get him to write down the number part of the registration plate. When we got home I'd get him to test both my memory and my mental arithmetic by asking me questions like "What number do you get if you add together the numbers of the 3rd, 17th, 18th and 43rd cars we passed and then take away the sum of the numbers of the 11th, 21st and 52nd cars". I never got one wrong.
Annoyingly though, my memory (and my skill at chess) was never as good as a guy I met when I was teaching. He could play, and beat, ten members of the junior county team at once without ever seeing the boards. (He had his back turned to them).