Dan Gardner is the New York Times best-selling author of Risk, Future Babble, Superforecasting (co-authored with Philip E. Tetlock), and How Big Things Get Done (co-authored with Bent Flyvbjerg). His books have been published in 26 countries and 20 languages. Prior to becoming an author, Gardner was an award-winning investigative journalist. More >

A Free Market In Good Judgement

They say the prime minister is a policy wonk with a keen appreciation for free markets. If that's true, I have an idea he's going to love.

One word: plastics.

No, sorry. That's a line from The Graduate. But my idea is almost as old as that movie.

Two words: prediction markets.

If you've read James Surowiecki's 2004 bestseller The Wisdom of Crowds, you know the basic idea.

Surowiecki illustrated it with the famous 1906 country fair attended by the renowned British scientist Francis Galton.

At the fair, there was a contest. People were invited to take a look at a hapless ox and answer a simple question: How much will this ox weigh after it has been "slaughtered and dressed"? The person who guessed closest to the mark would take home the carcass, a prize that was considerably more appealing in 1906.

Eight hundred people gave it a go. They included some experts, like butchers, but also plenty of people who wouldn't know which end of the cow eats the grass.

The ox's prepared carcass turned out to weigh 1,198 pounds. The winning guess was 1,170 pounds. But Galton asked the organizers to give him the tickets on which people had written their guesses and some simple math revealed that the aver-age guess - the collective judgment of all who took part - was 1,197 pounds.

The crowd was essentially perfect.

Or at least that's how this result is often framed. Personally, I don't like the word "crowd." A "crowd" suggests lots of people, often similar to one another, in the same physical space, talking together, sometimes noisily. That sort of crowd is subject to groupthink and other dynamics that make it anything but wise.

But the people guessing the doomed ox's weight in 1906 weren't that sort of crowd. They were diverse. They didn't discuss the issue as a group. They had no leader loudly declaring a certain choice to be the right one, no boss hinting it would be best if they got with the program. They were simply free to judge, individually, as best they could.

And the combined judgment of all those people was spot on.

There's no hocus-pocus involved here, only the principle that sol-id information is often widely distributed. One person knows a little something about the matter at hand. Another knows something else. And so on. If you collect and combine all that information, you will have a collective store of information that is greater than the in-formation held by any one person.

Of course, mistakes and nonsense are also distributed. But when judgments are aggregated, errors tend to cancel each other out.

Net result: a collective judgment that is far better than any one per-son's.

The same idea is at work in equity, commodity and currency markets, but its applications are limited only by imagination. In 1968 - the year after The Graduate was released - the U.S. Navy asked a diverse group to guess the location of a lost submarine. No individual came close. But the submarine was found only 220 yards from where the group's collective judgment said it would be.

In 1988, the Iowa Electronics Market asked investors to buy con-tracts on the outcome of the presidential election - in effect, to bet on the outcome - with the rising or falling value of the contracts measuring the collective judgment of the market about the candidate's prospects. With only 800 people involved, the IEM was actually more accurate than national polling conducted at great expense by major companies. In the years since, the IEM expanded and others opened, using a proliferating array of designs. They aren't perfect. But they're pretty darn good.

In the late 1990s, corporations started to create internal markets. Sales forecasts, shipping dates and stock prices. New ideas. Competitors' moves. Google is best known for pursuing these bottom-up initiatives, but as Donald N. Thompson shows in his recently published book Oracles (Harvard Business Re-view Press), many others have done the same. A few have even made internal markets the key source of their strategies and tactics.

The government can experiment with something similar.

Will a particular policy reduce crime? What will next year's deficit be? Will Greece default? Will greenhouse gas reduction targets be met? There is information about these sorts of questions scattered throughout the government. Markets could gather it up and deliver it to decision makers with vastly greater efficiency than any organization could muster, and the collective judgments that result would routinely be more accurate than those of even the smart and well-in-formed people on top.

But that's the problem, isn't it? For the people on top to create and consult markets, it takes a certain humility. They have to admit they don't know everything, and acknowledge that their judgment is not as good as the distilled wisdom of the many. They also have to be prepared to hear what they don't want to hear.

Flexible and creative organizations run by flexible and creative people - organizations like Google - can do that.

But rigid, hierarchical organizations where protecting the status quo is the primary mission? Organizations shaped like pyramids with a pharaoh perched on top? They can't do that.

No organization is more rigid and hierarchical than the federal government, and no leader in a modern liberal democracy is more like a pharaoh than the prime minister.

Which is why, despite six years in power, despite the fame of Surowiecki's book, despite Stephen Harper's affinity for markets, the Conservatives have never given this idea a try. And probably never will.

The problem isn't that it wouldn't work. The problem is that it would.