The person lip reading need not necessarily be deaf, but could still have the ability to read lips.
The person asking the question, "can you hear me", could be asking someone in a noisy environment. The person to whom the question is directed, may not have heard the question, due to the high noise levels, but could have read their lips, in which case they can answer the question with, "no".
Sorry, I don't care what the book says .... but it just doesn't work >.