On October 16, 2004, three press releases from the University College London (UCL) (Multimedia Appendix 1), Wiley InterScience in the United Kingdom, publishers of The Cochrane Library, and the Center for the Advancement of Health (CFAH) in the United States were widely disseminated to announce the result of a just-published Cochrane review synthesizing "studies on Internet health" (UCL press release) or, more accurately, interactive health communication applications (IHCAs) [1]. The Cochrane review seemed to arrive at stunning results that "confound conventional wisdom" (quote of the Principal Investigator, taken from the UCL press release): the Cochrane investigators found that these applications lead to an increase in knowledge and positive feelings of social support, but they had deleterious effects on health outcomes, that is, "may leave [users] in worse health" (UCL press release). These surprising results were immediately jumped on by the mass media and led to widely publicized news stories around the globe, with often sensationalistic and oversimplified headlines, such as "Internet Makes Us Sick," "Knowledge May Be Hazardous to Web Consumers' Health," "Too Much Advice Can Be Bad for Your Health," "Click to Get Sick?" and even "Is Cybermedicine Killing You?" (see Figure 1, Table 1, and Multimedia Appendix 2).

Figure 1

A collage of headlines reporting on the IHCA Cochrane review

Few reporters seemed to have read the review, which actually did not speak about health websites or "the Internet" (as suggested in the press release and subsequent media reports) but of "Interactive Health Communication Applications (IHCAs)." According to the Cochrane review, the defining feature of an IHCA is "that it does not simply provide health information, but combines such information with at least one (and frequently more than one) additional service [such as] decision support, behaviour change support or peer support" [1], which excludes information-only websites. Ignoring this, the press release spoke of the "Internet" and contained statements like "knowledge-seekers become so steeped in information from the Internet they make treatment choices on their own, contradicting advice from their doctors." Many news outlets reprinted the press release verbatim, which also stated that "people who use their computers to find health information often wind up in worse condition than if they had listened to their doctor," or rephrased this into "Some people with chronic health problems who seek online advice would be better off just listening to their doctors." Some journalists even condensed this to "Patient, don't try to heal thyself." Statements like this—emphasizing that people should better listen to their doctor rather than going on the Web—made consumers, patients, self-care advocates, and eHealth experts wince, not only because they seemed factually questionable, but also because they were reminiscent of a dark pre-Internet era of paternalism and "doctor knows it all" mindsets, which many thought were long behind us [2-6].

Most eHealth researchers are driven by the belief that the Web and other interactive media applications play a major role in supporting patients with chronic conditions. At the same time, we are all for being on the cautious side, viewing eHealth applications with a critical eye, knowing that some people will not benefit from them, stressing that badly designed applications can harm patients, and monitoring unintended side effects and potentially negative outcomes [7]. Still, many eHealth researchers were surprised and angered by the sweeping and blatant comments stemming from this review, which seemed to ignore the growing literature on the effectiveness of many eHealth interventions, some of which have been published in this journal. Most researchers familiar with the literature know that the vast majority of such reports are actually positive—in fact, the proportion of positive studies is so overwhelming that it has been questioned whether negative studies are underreported [8].

Figure 2

The original (flawed) figure from the retracted Cochrane review [1], showing the reverted effect estimates favoring the control rather than IHCAs

The series of unfortunate events culminated when the scientific eHealth community debunked the Cochrane review as a "methodological disaster" [9]. Among several other flaws (outlined below), the review included severe extraction (or coding and data interpretation) errors leading to a complete reversal of the outcomes. Positive outcomes in the primary studies (such as reduction in encopresis [10]) were misinterpreted as negative (harmful) effects. As Per Egil Kummervold and colleagues listed in the feedback section of the Cochrane Library on October 28, 2004, at least 8 of 11 outcomes were reversed—letting the effect estimator appear on the left side ("favours control") instead on the right side ("favours intervention"). See Figure 2, which shows the flawed figure from the Cochrane review, and Multimedia Appendix 2, which shows corrections made by Kummervold et al.

These were stunning errors because anyone who read these primary reports could not possibly have come to the conclusion that any of these studies reported less favourable health outcomes in the IHCA groups. To date, it remains a mystery how respected and experienced investigators could arrive at these conclusions (unless investigators relied on research assistants or students to extract the data and did not bother to read the studies themselves, which is an unimaginable scenario for a Cochrane review). These errors were obviously magnified by aggressive marketing efforts of the investigators and publisher, who sent out three press releases that did not in any way caution readers about the results.

When the review was eventually retracted by the authors on October 29, 2004—only 13 days after the press release—the public hardly took notice. As illustrated in an article by Roy Rada in this issue of JMIR [11], the media remained quiet, too quiet. To date, many publications have not published any follow-up stories, and the impact will be long-lasting. As the Rada paper shows, the Web is still full of reports on the flawed Cochrane review, and Rada identified only one newspaper story about the retraction—the Canadian journalist said he found this out only by chance. (We are also aware of a report by Frith Rayner, published in the Australian Doctor, courtesy of Lee Ritterband.) The failure of the media to report the retraction has to do with either the fact that they simply did not know about it or with issues around how the media decides what will be newsworthy ("if it bleeds, it leads," "bad news are good news"). Another reason why it was not taken up could be that the press release reporting the retraction was not very clear in highlighting the magnitude of the error, and it contained little more than the message that the review was being reworked and that it was too early to say what the result would be. Few journalists would have understood that the errors invalidated the results completely, even reversed them.

Rada analyzes the impact of the review and draws a few lessons, some of which shall be complemented by this editorial, not least because the authors of this editorial were involved in the events that eventually led to a retraction.

In a curious way, the media attention this review received—as detrimental as it was in sending out a false message to the public—also had a positive side in that it probably also sped up the identification of the errors. Without the media frenzy the results would possibly have remained unnoticed for a while, but with the worldwide media attention, peers quickly heard about the review.

For example, the editor of this journal (Gunther Eysenbach) was contacted by a journalist when the press release came out, was among the first who looked closely at the original review on October 16, and was one of the first who blew the whistle, pointing out that this emperor did not have clothes. Having read many of the primary studies that were pooled in the review, having done several systematic reviews in this area, and knowing the results of another review which JMIR published around the same time [12], he told the journalist who contacted him that the study seemed flawed. On October 25, he also posted a message on the Medical Webmasters Mailing List (MWM-L), where some researchers had started to discuss the study, warning readers not to take the review at face value [9].

Around the same time, on October 24, another researcher, Lee Ritterband—whose research was cited in the review and who had not seen a pre-print of it prior to publication—was alarmed by media reports and commented in a mailing list for Internet health intervention researchers: "While it is possible that some people may be worse off, we know that our interventions are quite effective, and this kind of fear-inducing 'findings' are the types of comments which our research, in part, must debunk." (Ritterband, personal communication, June 15, 2005). On the other side of the Atlantic, Per Egil Kummervold and colleagues at the Norwegian Centre for Telemedicine had also noticed that the numbers did not add up. They started a thorough investigation, reviewing the original data material, and on October 28 they notified the Cochrane Collaboration that the authors had made almost inconceivable mistakes, including reverting the direction of the results. On October 29, 2004, the review was retracted.

Fortunately, this particular Cochrane review warning patients to abstain from a specific type of intervention was not about a drug or other clinical intervention, whose withdrawal could have cost lives.

But damage was done: statements in the press release suggesting that "patients are better off listening to their doctor than going to the Internet" have outraged patient advocates (rightly so) and eroded the public's trust in the medical profession, which appeared to warn of the dangers of the Internet for selfish reasons. This was expressed in a posting by a patient on the BrainTalk forum, who wrote, "If the medical profession had its way this forum would be illegal" [35].

The myth of the Internet causing harm to your health may be here to stay, at least for a while, and policy makers and researchers searching the Web for evidence on the effectiveness of IHCAs will inevitably run into media remnants of the Cochrane review and cite it without bothering to read the original or corrected version. While Rada [11] failed to find any citation to the review in the Web of Science database (which is not surprising since in most traditional journals [not JMIR] it takes many months or years from submission to publication), one of the authors (GE) has already seen, as a peer reviewer, one book chapter and one thesis citing the Cochrane review without mentioning the retraction status.

It is our hope that by publishing this editorial and the Rada paper we do our part in making the public and the research community aware of this series of unfortunate events. While much of the damage created in this case is irreversible, lessons should be learnt so that future disasters can be avoided.