I think when Stephen Fry left they should have replaced Alan Davies as well, maybe not have a regular member at all.
A bit of QI trivia.
The original plan for QI was to have Michael Palin as the host and have a "bright" team and a "dim" team.
Stephen Fry would head the "bright" team and Alan Davies the "dim" team.
But Michael Palin said he did not want to do it, so when it came to record the pilot or "test" for the show they temporarily put Stephen Fry as the host.
And he was so good they stuck with him.
This is why you have one regular, Alan Davies, on one team but no regular on the other team.