Dan Gardner is the New York Times best-selling author of Risk, Future Babble, Superforecasting (co-authored with Philip E. Tetlock), and How Big Things Get Done (co-authored with Bent Flyvbjerg). His books have been published in 26 countries and 20 languages. Prior to becoming an author, Gardner was an award-winning investigative journalist. More >

Fie On Your Evidence-Ignoring Agenda!

Ideas come from a lot of places, and some of them are better than others. Is the patient sick? Stick some leeches on him to suck out the bad blood. That made sense to somebody, so they did it, and the patient felt better. It must have worked. Others tried it and sometimes the patient felt better, sometimes worse, which meant, naturally, that while the treatment didn't always work, it sometimes did, which is good. So more and more people did it, and pretty soon there were so many people sticking leeches on sick people it seemed obvious the treatment worked because they couldn't all be wrong. Nobody asked if maybe they were mistaking correlation for causation, or there was a placebo effect at work, or popularity isn't evidence of efficacy. They just kept sticking leeches on sick people. And when they had been doing it as long as anyone could remember, they kept on doing it because, well, that's what they had done for as long as anyone could remember. Needless to say, this is not a good way to develop medical treatments and we don't do it this way anymore. Now, we insist on proper scientific testing. But we do continue to use countless traditional treatments that were developed the old-fashioned way, which is why diligent medical researchers subject these treatments to proper scientific testing and encourage doctors to stop doing what they've always done if, in fact, it doesn't work. That's evidence-based medicine. Simple enough, isn't it? Now consider public policy. Politicians love evidence-based public policy. Evidence is essential, they say. It's what public policy should be based on. Why, it's a travesty when the other guys ignore the evidence and push their ideological agenda instead. "The approach of the Government of Canada must be based on scientific evidence," a Liberal motion on maternal health care recently declared. Fie on your evidence-ignoring ideological agenda, Stephen Harper! So, they all agree. Evidence is what counts. And they have evidence, oh yes they do. In a letter to the Globe and Mail, Liberal leader Michael Ignatieff said the long-gun registry works. "There has been a decline in all types of gun deaths since the registry was brought into force," he noted. See? We put the leeches on the patient. The patient felt better. So the treatment works. But the gun registry came into force in 1995. And after exhaustive research -- seven seconds on the Statistics Canada website -- I discovered that all types of gun deaths started falling in 1979. So the patient had been on the mend for 16 years before he got the leech treatment, which makes it somewhat less impressive that he continued to feel better after. Unfortunately, this is all too typical of the evidence on which politicians' evidence-based policy is based. There are plenty of anecdotes and testimonials, and the opinion of this or that authority figure is often treated as if it were the equivalent of a comprehensive trial at the Mayo Clinic. The fact that other countries believe leeches are the cutting edge is routinely considered persuasive, especially if one of those countries is the U.S. And we could wipe out the federal deficit within a year if we fined politicians $100 every time they presented post hoc nonsense like Michael Ignatieff's. But the very low bar politicians set for evidence only applies in certain circumstances. Namely, when the evidence supports a policy the politician favours for political or ideological reasons. When a politician opposes a policy, the bar rises considerably. Back in 2003, Rob Runciman, then Ontario's Conservative minister of public safety, demanded the auditor general determine the full cost of the gun registry and subject it to the gold standard of public policy research -- a rigorous cost-benefit analysis. Years later, having demonstrated such exacting standards and excellent judgment, Runciman was appointed to the Senate by Stephen Harper. At a press conference with the prime minister, Runciman said his immediate goal was to get the Senate to pass tough-on-crime legislation, which was mildly amusing given the tough-on-crime legislation has not been put to a cost-benefit analysis and would almost certainly fail miserably if it were. Enter the empirically minded Liberals. "We expect the government to be basing decisions on evidence," said Liberal critic Mark Holland, "as opposed to playing politics with emotions and trying to bully people into voting for things that don't work." At this point, I invite the reader to return to the paragraph above, in which I discuss Ignatieff's evidence-based decision on the gun registry. Afterward, return here and insert a scathing witticism. Please note this impressively stupid back-and-forth is based on newer policies. Old policies are almost never seriously scrutinized by anyone. Which is absurd because if evidence-based medicine has taught us anything it's that untested methods we have been using as long as anyone can remember may work about as well as sticking leeches on sick people. Half the Criminal Code dates from the 19th century, which is why our prostitution laws use language -- "bawdy houses," "living off the avails" -- not found outside the novels of Sir Arthur Conan Doyle. Anyone care to fund research examining the efficacy of those laws? How about "marihuana" policy? The whole war on drugs? No? No. If politicians were keen to ensure public policies are based on evidence, they would greatly boost the funding of social science researchers and call on them to measure and weigh every sacred cow. But they won't do that because the results of measuring and weighing evidence may not be in their political or ideological interest. Better to just keep applying leeches.