Dan Gardner is the New York Times best-selling author of Risk, Future Babble, Superforecasting (co-authored with Philip E. Tetlock), and How Big Things Get Done (co-authored with Bent Flyvbjerg). His books have been published in 26 countries and 20 languages. Prior to becoming an author, Gardner was an award-winning investigative journalist. More >

What Do You Mean, "Evidence-Based Policy"?

Announcing his candidacy for the Liberal leadership, Justin Trudeau got a round of applause when he declared that, in seeking solutions to problems, "the only ideology that must guide us is evidence. Hard, scientific facts and data. It may seem revolutionary in today's Ottawa, but instead of inventing the facts to justify policies, we will create policy based on facts."

Personally, I did not have to suppress the urge to stand and cheer.

Sure, it was nice to hear a politician identify and address the issue. But Trudeau said nothing that suggests he understands the problem. In fact, what he said strongly hints that he really doesn't get it.

Start with that comment about "ideology." Like a lot of people, Trudeau doesn't know what the word means.

Ideology is simply "a system of ideas or way of thinking," as the OED puts it. It is the lens through which someone perceives reality. And when I say "someone," I mean you, me, and everybody else. We all possess a more or less coherent ideology. We couldn't function without one. Sometimes we are aware of our ideology and think carefully about it, but more often it is implicit in our thinking and we give it no more thought than a fish gives to the water it breathes.

Of course what Trudeau meant to say, but did not, is that if and when ideology conflicts with evidence, he will go with evidence.

That would indeed be a change from the way Stephen Harper operates: When the prime minister confronts a conflict between ideology and evidence, he tosses both in a Dumpster and does what's best for Stephen Harper.

But saying you'll insist that policies be supported by evidence isn't nearly as meaningful as Trudeau and other Liberals seem to think it is. Because evidence comes in all shapes in sizes.

In many cultures around the world, it was traditionally believed that eclipses are caused by some giant creature - a dragon, jaguar or monkey - trying to eat the celestial object.

The obvious solution was to make a great deal of noise to drive off the creature. And it worked. After people made noise, the eclipse ended. Every time.

That's evidence.

When Uncle Frank feels so much better after his appointment with the naturopath, that's evidence.

When the government passes new mandatory minimum sentences and crime goes down, that's evidence.

It's all evidence. It's crappy evidence. It's evidence that doesn't really prove anything. But it's evidence.

Those are simple examples, but people are capable of producing the most amazing quantities of fantastically complicated and impressively detailed evidence in support of claims that are completely wrong.

Read Fads and Fallacies in the Name of Science, Martin Gardner's classic examination of pseudo-science (no relation, alas), and you will quickly realize that even the most ridiculous quack can bury you in evidence.

This helps explain why you never hear a politician say, "My policy is supported by absolutely no evidence whatsoever." Their policies are always supported by evidence. Very often it's crappy evidence. But it is evidence.

A couple of years ago, then-Liberal leader Michael Ignatieff claimed the long-gun registry works because "there has been a decline in all types of gun deaths since the registry was brought into force" in 1995 - without mentioning that this decline was merely a continuation of a trend that started in 1979. That was evidence. It was about as strong as the evidence that suggests loud noises keep dragons from swallowing the sun. But it was evidence.

Same for Mark Holland's response when I asked the former Liberal justice critic, on Twitter, to point me to what he had said was the abundant evidence that proved the gun registry's effectiveness. Holland noted that several respected organizations backed the registry. That was evidence. Of a sort.

Politicians treat flimsy evidence as if it were granite when it supports what they want supported but they ignore or dismiss evidence that is anything less than randomized, double-blind, Nobel Prize-winning perfection when it does not. That's understandable. Everyone tends to think this way, not just politicians. It's called "confirmation bias."

This tendency is precisely why, if Justin Trudeau and the Liberals are serious about making policy evidence-based, they won't promise evidence-based policies. Instead, they will promise the production of top-quality research.

Make randomized trials and cost-benefit analyses of new policies universal and mandatory. Create a system for systematically subjecting existing programs to the new standards.

Ensure that every scrap of evidence is made freely available to all in a single information clearing house.

Also, create the Canadian equivalent of the United States Government Accountability Office.

The best way to do that may be to greatly expand the remit and budget of the auditor general or the parliamentary budget officer. Or perhaps it would be best to create a separate "evaluator general," as some have suggested.

In addition, there needs to be far greater support for independent policy analysis in universities and think-tanks.

All that will cost a lot of money. Pay it. This country suffers from a dearth of serious policy analysis in part because governments - Conservative and Liberal - have treated it as a frill and funded it accordingly. It's not a frill. It's essential.

It won't do away with confirmation bias. But a large and growing supply of top-quality evidence produced by credible sources would improve public discussion and make it increasingly difficult for politicians to peddle nonsense supported by piffle.

That's the evidence I need to see before I take the Liberals' talk of evidence seriously.