Well, it's Oxford, what else would you expect?
I'd always assumed that the main point of people moaning about this is that it shouldn't matter who are up on the walls, in which case it's always odd that any attempt to add diversity to the paintings is met with cries of anguish.
And another thing -- a fair number of the portraits are anyway of people whose main claim to fame is that they (a) went to the college and/or (b) gave a lot of money to it at some point. I doubt most people will have heard of most of the men in the portraits at King's, for example (or any other college for that matter). There was one of Walsingham, chief spy to Elizabeth 1st, and another of Robert Walpole, but most of the others I'd never heard of. Old provosts but that was about it.
So what I'm saying is that it's not obvious to me why you'd think that the portraits that are already there deserve to stay there enough for it to be so outrageous to suggest adding new ones that reflect the more modern diversity of the college.
Having said that, I think the problem is that reflecting heritage is often confused with celebrating it. That there are so many dead white men on the walls is because that's just how things were back then. I'm not sure it should be seen as offensive, but it shouldn't be so important to people to keep things that way either.