I would say, strictly speaking, the fickle nature of all TV viewers. The US figures are just the only ones I could find instantly. I expect it's possible to find estimates for global audience figures for various matches, and a similar pattern will be seen, viz. a massive variation in viewing figures that can't be attached to whether it's men or women playing, or at least not solely. The UK audience for the 2013 Wimbledon final hit something like 17 million or so -- and then 10 million next year, when Andy Murray wasn't in it; and 9 million the year after that, when he wasn't playing either.
Big names attract big audiences. But then of course the reason these players are big names is because they win the big tournaments in the first place, which in turn means they are already winning more prize money.
While I can see the argument Djokovic is making, and that others here in this thread support, the fact is that to say men's matches get bigger audiences than women's matches is a generalisation that doesn't hold true in all cases. If we are going to link prize money to ticket sales directly at all, it should at least be based on far more specific factors than just the gender of the player. I don't see how that's realistic. Because this is, after all, a tournament prize and not actually work-related pay.
I do think there's a separate case for encouraging more five-set tennis in women's matches, but that's because it would be more entertaining (in some cases at least*), rather than on equality grounds.
*but then not all men's matches are equally entertaining either just by virtue of being longer; quite a few matches are one-sided affairs, or close but not particularly high-quality.