This editorial briefly reviews the series of unfortunate events that led to the publication, dissemination, and eventual retraction of a flawed Cochrane systematic review on interactive health communication applications (IHCAs), which was widely reported in the media with headlines such as "Internet Makes Us Sick," "Knowledge May Be Hazardous to Web Consumers' Health," "Too Much Advice Can Be Bad for Your Health," "Click to Get Sick?" and even "Is Cybermedicine Killing You?" While the media attention helped to speed up the identification of errors, leading to a retraction of the review after only 13 days, a paper published in this issue of JMIR by Rada shows that the retraction, in contrast to the original review, remained largely unnoticed by the public. We discuss the three flaws of the review, which include (1) data extraction and coding errors, (2) the pooling of heterogeneous studies, and (3) a problematic and ambiguous scope and, possibly, some overlooked studies. We then discuss "retraction ethics" for researchers, editors/publishers, and journalists. Researchers and editors should, in the case of retractions, match the aggressiveness of the original dissemination campaign if errors are detected. It is argued that researchers and their organizations may have an ethical obligation to track down journalists who reported stories on the basis of a flawed study and to specifically ask them to publish an article indicating the error. Journalists should respond to errors or retractions with reports that have the same prominence as the original story. Finally, we look at some of the lessons for the Cochrane Collaboration, which include (1) improving the peer-review system by routinely sending out pre-prints to authors of the original studies, (2) avoiding downplay of the magnitude of errors if they occur, (3) addressing the usability issues of RevMan, and (4) making critical articles such as retraction notices open access.
If you are interested in stories with happy endings, you would be better off reading some other story. In this story, not only is there no happy ending, there is no happy beginning and very few happy things in the middle.
On October 16, 2004, three press releases from the University College London (UCL) (
A collage of headlines reporting on the IHCA Cochrane review
Epidemic of misinformation: selected headlines from around the world reporting the results of the flawed Cochrane review, compiled in October 2004 (as of June 2005, most of these articles are still online, and not a single one carries a note on the retraction)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Few reporters seemed to have read the review, which actually did not speak about health websites or "the Internet" (as suggested in the press release and subsequent media reports) but of "Interactive Health Communication Applications (IHCAs)." According to the Cochrane review, the defining feature of an IHCA is "that it does not simply provide health information, but combines such information with at least one (and frequently more than one) additional service [such as] decision support, behaviour change support or peer support" [
Most eHealth researchers are driven by the belief that the Web and other interactive media applications play a major role in supporting patients with chronic conditions. At the same time, we are all for being on the cautious side, viewing eHealth applications with a critical eye, knowing that some people will not benefit from them, stressing that badly designed applications can harm patients, and monitoring unintended side effects and potentially negative outcomes [
The original (flawed) figure from the retracted Cochrane review [
The series of unfortunate events culminated when the scientific eHealth community debunked the Cochrane review as a "methodological disaster" [
These were stunning errors because anyone who read these primary reports could not possibly have come to the conclusion that any of these studies reported less favourable health outcomes in the IHCA groups. To date, it remains a mystery how respected and experienced investigators could arrive at these conclusions (unless investigators relied on research assistants or students to extract the data and did not bother to read the studies themselves, which is an unimaginable scenario for a Cochrane review). These errors were obviously magnified by aggressive marketing efforts of the investigators and publisher, who sent out three press releases that did not in any way caution readers about the results.
When the review was eventually retracted by the authors on October 29, 2004—only 13 days after the press release—the public hardly took notice. As illustrated in an article by Roy Rada in this issue of JMIR [
Rada analyzes the impact of the review and draws a few lessons, some of which shall be complemented by this editorial, not least because the authors of this editorial were involved in the events that eventually led to a retraction.
In a curious way, the media attention this review received—as detrimental as it was in sending out a false message to the public—also had a positive side in that it probably also sped up the identification of the errors. Without the media frenzy the results would possibly have remained unnoticed for a while, but with the worldwide media attention, peers quickly heard about the review.
For example, the editor of this journal (Gunther Eysenbach) was contacted by a journalist when the press release came out, was among the first who looked closely at the original review on October 16, and was one of the first who blew the whistle, pointing out that this emperor did not have clothes. Having read many of the primary studies that were pooled in the review, having done several systematic reviews in this area, and knowing the results of another review which JMIR published around the same time [
Around the same time, on October 24, another researcher, Lee Ritterband—whose research was cited in the review and who had not seen a pre-print of it prior to publication—was alarmed by media reports and commented in a mailing list for Internet health intervention researchers: "While it is possible that some people may be worse off, we know that our interventions are quite effective, and this kind of fear-inducing 'findings' are the types of comments which our research, in part, must debunk." (Ritterband, personal communication, June 15, 2005). On the other side of the Atlantic, Per Egil Kummervold and colleagues at the Norwegian Centre for Telemedicine had also noticed that the numbers did not add up. They started a thorough investigation, reviewing the original data material, and on October 28 they notified the Cochrane Collaboration that the authors had made almost inconceivable mistakes, including reverting the direction of the results. On October 29, 2004, the review was retracted.
The most devastating (and most obvious) error was the previously discussed blatant mistake of misinterpreting positive outcomes as negative outcomes (and vice versa). Kummervold's original list of errors is documented in
The second concern is that the studies were too heterogeneous to sensibly attempt a formal random effects meta-analysis using Cochrane's RevMan software. The resulting effect estimates are meaningless. The review guidelines of the Consumers and Communication Group state the following: "They [systematic reviewers] should also use caution when extracting and interpreting data, and when deciding whether to combine them statistically. Combining disparate data quantitatively may not always be appropriate, and qualitative synthesis may often be preferable" [
This is particularly true for pooling health outcomes, but equally problematic is to pool knowledge scores and behaviour change and social support measures from studies with interventions that had very little in common except that they were delivered electronically. Comparing different IHCAs against controls and pooling these results without paying any attention to the "ingredients" is like comparing all studies in which investigators used "blue pills" in the intervention arm—a pooling on the basis of the delivery mechanism rather than the "ingredient" is of limited value. And for every successfully IHCA-delivered intervention, one can probably find a similar intervention which is delivered on a badly designed IHCA, which does not mean that IHCAs per se are inappropriate delivery mechanisms.
A richer, deeper qualitative analysis to answer questions like "what seem to be the success factors in terms of how the intervention and the trial should be designed" would have been more appropriate and more informative. Qualitative synthesis prevents us from drowning in a river that on average is only 3 feet deep.
As an aside, while the authors focused on extracting outcome measures which could support their postulated pathway of action, it would have been very informative to extract and report attrition rates as secondary outcomes (ie, the percentage of users who dropped out and/or did not use the application), not only because nonuse of the application may explain a lack of an effect, but also because such data from numerous studies could be useful in identifying some of the factors (predictors) for nonuse/dropout as postulated in the "Law of Attrition" [
The third problem with the IHCA Cochrane review, which has not yet been discussed on the Cochrane feedback section, is the scope of the review and the lack of truly comprehensive searches within the scope the authors defined. The scope may be too broad in some respect (making the review unmanageable and confusing by lumping together too many different applications), and too narrow in others (eg, by excluding pure patient-doctor or peer-to-peer communication, or by focusing on chronic diseases).
In particular, the authors decided to exclude electronic decision aids and computer/Internet-delivered cognitive behavioural therapy (CBT) programs. High-quality CBT applications, such as ODIN (Overcoming Depression on the InterNet), were not cited in the review [
There is also some confusion about the scope of the review, in particular, whether applications that provide social support by enabling peer-to-peer communication were included. The original definition of the Science Panel on Interactive Communication and Health [
[IHCAs are] the operational software programs or modules that interface with the end user. This includes health information and support Websites and clinical decision-support and risk assessment software (which may or may not be online), but does not include applications that focus exclusively on administrative, financial, or clinical data, such as electronic medical records, dedicated clinical telemedicine applications, or expert clinical decision-support systems for providers.
However, the Cochrane review team narrowed this definition by only including studies on applications that, apart from delivering health information, had another component (eg, decision support, peer-to-peer support), thereby excluding simple information-only websites. The fact that simple health information sites were not in the scope of the review was not communicated properly in the press release and was widely misunderstood by the media. Journalists reported, for example, that "the study found no evidence that Web health information [
On the latter point, the definition of IHCA used by the Cochrane review team leaves considerable ambiguity about whether or not "pure" peer-to-peer groups are in the scope of the review—and ambiguities at the protocol stage are often a recipe for disaster [
A final concern is that a comparison with another systematic review [
In the "publication ethics" literature, which deals with scientific misconduct such as duplicate publication, underreporting, and authorship issues, there is remarkably little discussion on how retractions due to error or misconduct should be handled by investigators and the media. In the case discussed here, the Cochrane Collaboration and the investigators were, as the Rada paper [
Similarly, in our view there is an ethical duty for journalists to respond to such requests and to react to reports on errors or retractions with stories that have the same prominence as the original story. In other words, if the original report was worth a space on the title page, the retraction should be reported on the same prominent spot. In cases for which it is possible to change or add something to the original story (online articles), this should also be done.
The Cochrane press release that was issued when the original report was retracted contained the following statement: "The Cochrane Collaboration regrets that this particular review was found to contain inaccuracies, apologises unreservedly, has acted swiftly to mitigate both this error (which arose from individual error and not systemic failures) and the likelihood of it being repeated, and undertakes to ensure that the corrected results are published as soon as possible" (Cochrane press release).
What is interesting here is that it took the Cochrane Collaboration only a few days to determine that there were no "systemic failures," which, in our view, is questionable. Perhaps a better approach would have been to set up an independent group to analyze the mistakes made and to wait for them to come back with some recommendations, rather than swiftly dismissing any possibilities for systemic errors.
One remarkable and obvious "systemic" problem seems to be the apparent total failure of the pre-publication peer-review system. Most eHealth researchers (and certainly those whose work was cited in the review) state that it took them only minutes to figure out that something was wrong with the review, which suggests that the 4 peer reviewers who reviewed the manuscript were not intimately familiar with the work done in this area. One potential policy change that the Cochrane Collaboration may have to make is a
As noted above, the Cochrane Collaboration and the investigators have not been successful in getting the word out about the error in a timely manner. It is not sufficient to wait for a corrected version to appear (which was promised for April 2005), hoping that the media and the public will remember the original story and correct their impressions of it. The press release issued by Cochrane seems to downplay the severity of the errors. It does not say that the errors were so grave that they literally led to a reversal of the conclusions, even though it was clear to any informed observer that the initial message Cochrane disseminated was the 180-degree opposite of what should have been reported.
The admittance of an error was half-hearted, and the marketers at Cochrane tried to use even the retraction press release as an opportunity to emphasize how good Cochrane reviews are compared with non-Cochrane reviews: "It has been demonstrated that Cochrane Systematic Reviews are of comparable or better quality and are updated more often than the reviews published in print journals" [
Another issue Cochrane should carefully look at is the usability of RevMan, the software used to support meta-analyses. From the experience of one of the authors (GE), RevMan clearly has some usability issues, most notably that it is far too easy to accidentally "flip" the direction of outcomes. This may have been a contributing factor to the errors in this case. The principal investigator wrote in the Cochrane Communication Consumer and Communication Group newsletter that "RevMan has a mind of its own and I don't think I could have managed it without our very own IT whizzkid,...the lead research fellow on the review" [
The Cochrane Library is (amazingly) still not an Open Access publication. This may have been a contributing factor to why the retraction remained largely unnoticed by the public and many fellow researchers. The UCL press release (
The "Reason for Withdrawal" behind closed doors—only subscribers have the privilege of learning about the retraction (as of May 30, 2005)
Fortunately, this particular Cochrane review warning patients to abstain from a specific type of intervention was not about a drug or other clinical intervention, whose withdrawal could have cost lives.
But damage was done: statements in the press release suggesting that "patients are better off listening to their doctor than going to the Internet" have outraged patient advocates (rightly so) and eroded the public's trust in the medical profession, which appeared to warn of the dangers of the Internet for selfish reasons. This was expressed in a posting by a patient on the BrainTalk forum, who wrote, "If the medical profession had its way this forum would be illegal" [
The myth of the Internet causing harm to your health may be here to stay, at least for a while, and policy makers and researchers searching the Web for evidence on the effectiveness of IHCAs will inevitably run into media remnants of the Cochrane review and cite it without bothering to read the original or corrected version. While Rada [
It is our hope that by publishing this editorial and the Rada paper we do our part in making the public and the research community aware of this series of unfortunate events. While much of the damage created in this case is irreversible, lessons should be learnt so that future disasters can be avoided.
PEK wishes to thank Professor Per Hjortdahl for assistance in analyzing the source material of the Murray review and in communicating the errors found to the Cochrane Collaboration.
Both authors have been involved in this case as whistleblowers. GE used to be involved in the Cochrane Consumers and Communication Group as the Cochrane Skin Group.
UCL press release
Errors in outcomes
Potentially missed studies
Media responses
interactive health communication application