Don't Watch Cable News

Dice. Coins. Roulette wheels. Monkeys throwing darts. Slips of paper in a hat. Eenie meenie miney mo. These are all methods of picking alternatives that would outperform the vast majority of political pundits. In my latest Wall Street Journal column, I expand on why, exactly, pundits are so terrible, and how we can do a better job of ignoring those who are almost certainly spouting telegenic nonsense.

It's an American tradition__:__ In the final weeks before an election, the airwaves are saturated with pundits and their bold predictions. This time around, they might be forecasting a decade of tea-party dominance, or the imminent comeback of the Democrats or a return to recession in the face of political deadlock. And as these pundits rattle off their reasons, they sound as if they know what they're talking about.

But do they? Philip Tetlock, a psychologist at the University of California, Berkeley, has spent 25 years trying to find out. He first got interested in the subject during the run-up to the 1984 presidential election, when dovish experts said Ronald Reagan's tough talk to the Soviets was needlessly antagonizing them, while hawkish experts were convinced that the Soviets needed to be aggressively contained. Mr. Tetlock began to monitor their predictions, and a few years later, he came to a sobering conclusion: Everyone was wrong. Both hawks and doves failed to anticipate the rise of Mikhail Gorbachev and glasnost, even if the pundits now claimed to have seen it coming all along.

The dismal performance of the experts inspired Mr. Tetlock to turn his case study into an epic experimental project. He picked 284 people who made their living "commenting or offering advice on political and economic trends," including journalists, foreign policy specialists, economists and intelligence analysts, and began asking them to make predictions. Over the next two decades, he peppered them with questions: Would George Bush be re-elected? Would apartheid in South Africa end peacefully? Would Quebec secede from Canada? Would the dot-com bubble burst? In each case, the pundits rated the probability of several possible outcomes. By the end of the study, Mr. Tetlock had quantified 82,361 predictions.

How did the experts do? When it came to predicting the likelihood of an outcome, the vast majority performed worse than random chance. In other words, they would have done better picking their answers blindly out of a hat. Liberals, moderates and conservatives were all equally ineffective. Although 96% of the subjects had post-graduate training, Mr. Tetlock found, the fancy degrees were mostly useless when it came to forecasting.

The main reason for the inaccuracy has to do with overconfidence. Because the experts were convinced that they were right, they tended to ignore all the evidence suggesting they were wrong. This is known as confirmation bias, and it leads people to hold all sorts of erroneous opinions. Famous experts were especially prone to overconfidence, which is why they tended to do the worst. Unfortunately, we are blind to this blind spot: Most of the experts in the study claimed that they were dispassionately analyzing the evidence. In reality, they were indulging in selective ignorance, as they explained away dissonant facts and contradictory data. The end result, Mr. Tetlock says, is that the pundits became "prisoners of their preconceptions." And their preconceptions were mostly worthless.

What's most disturbing about Mr. Tetlock's study is that the failures of the pundit class don't seem to matter. We rely on talking heads more than ever, even though the vast majority of them aren't worth their paychecks. Our political discourse is driven in large part by people whose opinions are less accurate than a coin toss.

Mr. Tetlock proposes forming a nonpartisan center to track the performance of experts, just as we track the batting averages of baseball players. In the meantime, he suggests that we learn to ignore those famous pundits who are full of bombastic convictions. "I'm always drawn to the experts on television who stumble a little on their words," he adds. "For me, that's a sign that they're actually thinking about the question, and not just giving a canned answer. If an expert sounds too smooth, then you should probably change the channel."

As Mr. Tetlock points out, the future is impossible to predict. Even with modern polling, we can barely anticipate the outcome of an election that is just a few days away. If a pundit looks far beyond that time horizon, to situations with a thousand variables and very little real information to back up a prediction, we should stop listening and get out a quarter.

I don't have much to add to the cable news critique of Jon Stewart, but I did want to spend a little more time on why pundits are so inaccurate. (It really takes a little work to consistently perform worse than random chance.) Although the study is best known for its demonstration of expert failure, Tetlock also found that a few performed far above average. He explained the difference between successful and unsuccessful pundits with an allusion to an ancient metaphor, made famous by Isaiah Berlin, who distinguished between two types of thinkers: hedgehogs and foxes. A hedgehog is a small mammal covered with spines that, when attacked, rolls itself into a ball, so that its spines point outwards. This is the hedgehog’s only defense. A fox, on the other hand, doesn’t rely on a single strategy when threatened. Instead, it adjusts its strategy to fit the particulars of the situation. Foxes are also cunning hunters. In fact, they are one of the hedgehog’s few predators.

According to Tetlock, the problem with pundits who think like hedgehogs is that they are prone to bouts of certainty - their big idea is irrefutable - and this certainty causes them to misinterpret the evidence. Useful information is deliberately ignored. Successful pundits, on the other hand, think like foxes. While hedgehogs reassure themselves with certainty, foxes rely on the solvent of doubt. They are skeptical of grand strategies and unifying theories. They accept ambiguity and take an ad-hoc approach when coming up with explanations. They gather data from a wide variety of sources. The end result is that foxes make better predictions and decisions.

But being open-minded isn’t enough. Tetlock found that the most important difference between foxes and hedgehogs is that foxes are more likely to study their own decision-making process. According to Tetlock, such introspection is the best predictor of good judgment. Because foxes pay attention to their inner disagreements, they are less vulnerable to the seductions of certainty. Tetlock says it best in his book, Expert Political Judgment: “We need to cultivate the art of self-overhearing, to learn how to eavesdrop on the mental conversations we have with ourselves.”