"Tose [sic] that are academically gifted enough and passed the interview are there."
It also helps to have the right support. I was perhaps on the threshold of getting that support, but was rather lucky. My school gave me two mock interviews; one, by the headteacher, ended up being worse than useless as preparation, because he was apparently stuck with the illusion that the interview was going to ask me questions like "why do you want to come here?" or "what else do you do?". Thankfully, my science teacher also gave a mock interview, asking, if I remember correctly, two questions, about oscilloscopes and streetlights. Even with that help, I still crashed and burned in the first interview. At any rate, anybody without at least a modicum of understanding as to what to expect is likely to be doomed to failure, regardless of how good they are. If you walk in expecting one interview and end up having another one entirely, then it's difficult to get your head together (perhaps this is why they often offer two interviews, to give everyone a second chance*).
This matters because, again, the resources on offer to those from public schools (mostly white people, remember) are vastly greater than for the schools that many black children end up attending. Eton, Harrow, Westminster, etc, have, as I mentioned, centuries of experience with the Oxbridge process to draw on to help guide their students through exams, interview techniques, and the like. By contrast, an under-resourced school with no historic links might have little if anything to draw on to support students that would otherwise have the talent needed, if only they were given the opportunity to let it shine through.
All of this boils down to one point: there can be no argument whatsoever that, at the end of it all, Oxbridge can, and must, select only on talent. But you are deluding yourself if you think that the outcome of this is fair, or has only to do with the ability of the student. Their background, their school, the support they receive, all matter, and right now these are unfortunately, and unacceptably, skewed against black people.
*There's a fun story about this, too, but at some point I think people will get bored of my anecdotes.