In Kurt Vonnegut's novel Galapagos, big-brained humans blow up the world with nuclear weapons. The only survivors are cruise-ship passengers shipwrecked on one of the Galapagos Islands of Darwin fame.
Survival of the fittest plays out on the island, with those able to catch fish better suited to eat, live, mate, and pass on their genetic information.
Smart people--the kind who can build weapons that destroy the world--are at a disadvantage on the island because all they know how to do is argue. They all soon die. The dumb people, over the course of millions of years, evolve into dumber, penguin-like creatures skilled at catching fish.
Vonnegut clearly doesn't have much respect for those with big brains. By "big brain" of course, he means the so-called smart person--creative liberty from a great author who knows deep down that HUMAN BRAIN SIZE HAS NOTHING TO DO WITH INTELLIGENCE.
Assuming you could measure smartness (which we can't), and assuming you could measure brain size by measuring the outside of the head (which we can't), you'd still be wrong to assume that people with bigger heads are smarter. There have been geniuses with tiny brains and idiots with huge ones.
Women have smaller brains than men, on average. Smaller people, particular midgets, often have smaller brains. Unless you are prepared to defend the stance that women and short people are dumber, you'd be wise to drop the "BIG BRAIN = BIG SMARTS" argument.