In the past few weeks, the blogosphere has been abuzz about the dangers of non-replication and the "decline" effect, triggered by Jonah Lehrer's interesting piece in the New Yorker (mostly behind a firewall). The central claim in the piece is that initially strong or provocative findings diminish in strength over time. The decline might well come from more stringent methodology or better experimental controls rather than via mysterious forces, but that's not what concerns me today.

My concern is about media reporting and even blogging about new and provocative scientific findings, the very findings that tend to decline. Following a murder, the arrest of a suspect is broadcast on the front pages, but when that suspect is exonerated, the correction ends up on the back of the local section months later (if it appears at all). The same problem holds for flawed scientific claims. The thoroughly debunked Mozart Effect still receives media coverage, just as other unsupported findings remain part of the popular consciousness despite a lack of replicability.

Part of the problem is the rush to publicize unusual or unexpected positive findings, particularly when they run counter to decades of established science. That excitement about a new result is palpable and understandable. Who wants to write about the boring old stuff? The media loves controversy, and new results that counter the establishment are inherently interesting. Scientists strive for such controversy as well—what scientist doesn't relish the idea of overhauling an accepted theory?

Scientists understand that initially provocative claims don't always hold up to scrutiny, but media coverage rarely withholds judgment. If well-established ideas can be shot down by a single study, and that single study gets extensive media coverage, the public understandably won't know what to trust. The result, from the perspective of a consumer of science, is that science itself appears unstable. It gives people license to doubt non-controversial claims and theories (e.g., evolution). To the public eye, a single contradictory study has the same standing as established theory.

Over the past few days, a paper in PLoS has received extensive attention in the media and on science blogs. The paper reports a study in which patients showed a placebo effect even when they knew they were receiving a placebo. If true, the result would undermine the idea that placebos are effective because people think they are getting the real treatment. The result is shocking and intriguing. It inspired headlines like "Placebos Work Even When You Know and "Sugar Pills Help, Even When Patients Are Aware of Them."

The study is small in scope (80 patients), and some bloggers have already begun raising concerns about the method (e.g., Orac, Ed Yong). The bigger issue, though, is that the paper runs counter to long-established theories about the nature of placebo effects. That alone should inspire caution rather than exuberance. This one study, essentially a pilot study, should not lead anyone to reject a long-established empirical tradition. Sure, it can raise questions about the established idea, and it should trigger further research with larger samples and alternative methods. Critically, scientists know that new claims like this one are more likely to "decline" with replication than are well-established results, and they know that such preliminary results require further study. The media, though, gives the same weight to a pilot study like this one as to a larger body of research. Controversial results are reported as the new truth, meaning that scientific "facts" change with each new study.

When facts are so easily undermined in the public presentation of science, the public justifiably distrusts scientific claims. Ironically, conveying uncertainty when reporting new results, particularly those that run counter to well-established findings, might increase the public's confidence in science over time. Acknowledging the tentativeness of new findings avoids the danger of having the "facts" change with each new result. It avoids having the truth wear off.

Sources Cited:

Chabris CF (1999). Prelude or requiem for the 'Mozart effect'? Nature, 400 (6747), 826-827 PMID: 10476958

Kaptchuk, T. J, Friedlander, E., Kelley, J. M., Sanchez, M. N., Kokkotou, E., Singer, J. P., Kowalczykowski, M., Miller, F. G., Kirsch, I., & Lembo, A. J. (2010). Placebos without Deception: A Randomized Controlled Trial in Irritable Bowel Syndrome PLoS One, 5 (12)