In a recent article for The Atlantic, David Freedman told the story of Dr. John Ionnidis, who for years has been at the forefront of medical "meta-research"--research about research. Ionnidis has shown that vast numbers of studies are simply bogus--poorly designed, biased by a need for flashy results, and so forth. The efficacy even of well-established drugs like Prozac may have been "established" with faulty research, and we know far less about the effects of sun exposure than we think we do. To add to that, explains Freedman:
Much, perhaps even most, of what doctors do has never been formally put to the test in credible studies, given that the need to do so became obvious to the field only in the 1990s, leaving it playing catch-up with a century or more of non-evidence-based medicine ... That we’re not routinely made seriously ill by this shortfall, [Ionnides] argues, is due largely to the fact that most medical interventions and advice don't address life-and-death situations, but rather aim to leave us marginally healthier or less unhealthy, so we usually neither gain nor risk all that much.There's a less than reassuring thought: your doctor's prescription may not be hurting you, simply because it may not be doing much of anything. Here's some of the response to Ionnidis's provocative thesis.
- The Power of Wild Claims "The $300,000 kindergarten teacher, anyone?" offers economist Arnold Kling, referring to a study that suggested that figure as the "worth" of a competent early educator. He's responding to a passage in the article describing how 'extravagant' findings in medical science tend to take hold in the public imagination, even if they're not necessarily true: "it's that very extravagance of claim ... that helps gets these findings into journals and then into our treatments and lifestyles, especially when the claim builds on impressive-sounding evidence"
- Okay, But I'll Take Medical Science Over No Medical Science Sure, medical science can be poor, says Jim Manzi at National Review and The American Scene. "But as always, the question is, 'Compared to what?'" The impact of imperfect progress is still probably a net positive: "medical research is not the only scientific field in which funding pressure creates researcher biases, yet we still seem to be able to build functioning airplanes and mobile phones." Also, he points out that the studies mentioned in the article "tend very strongly to be long-term, behaviorally-oriented interventions" and adds that, more generally, results can conflict when a problem is simply hard to solve. In other words: "The problem isn't always the researchers; sometimes the problem is the problem."
- Shows Medical Science Not Necessarily Better than Social Science, says Thorfinn at Gene Expression, referring back to an earlier essay
by Manzi contending "the Social Sciences were so far behind the hard
sciences because of the problem of causal density. Without the benefits
of randomization and experimentation available in the physical
sciences, it’s hard to figure out causality--or so Manzi argues." The
point of this new article, Thorfinn says, is that "having recourse to
randomization isn’t sufficient to generate knowledge. The real problem
there is experimenter bias."
- Well, for Starters, Just Get a Second Opinion, concludes George Mason economist Robin Hanson,
who says he was "glad" the article "illustrated how ... skepticism
informs the practice of a clinician on Ioannidis' team." For example: a
physician performing a careful second exam.