Yes, gravity at the centre of the Earth is zero. Strictly speaking this is only true if there were no other stars and planets in the Universe but what's meant is that the force due to Earth specifically cancels at the centre (as everything is pulling you equally in all directions, leading to no force overall).
If it suddenly became "infinity", that can occur only if there were a literal object of zero size at the centre of the Earth, and would only be meaningful if you were also an object of zero size sitting right on top of it, so that there was exactly no distance between you and it.
This is not really meaningful physically. But if we scale things up very slightly, and allow for a very large, small and dense object sitting at the centre of the Earth, and allowed yourself to get closer and closer to it, then -- well, then congratulations, you've just discovered Black Holes.
Even then, the force at the centre of a Black Hole would still be zero, although at that point it's not clear what the question even means because space and time at the centre of a Black Hole are so messed up as to be meaningless.
* * * * * * * * * *
You can also show that the gravitational force everywhere inside a spherical "shell" is zero.