Published on 28.07.05 in Vol 7, No 4 (2005)
"Is Cybermedicine Killing You?" - Peer Review and Evidence-Based Medicine
Comment on: Eysenbach G, Kummervold PE. "Is Cybermedicine Killing You?" - The Story of a Cochrane Disaster. J Med Internet Res. 2005;7(2) p. e21 http://www.jmir.org/2005/2/e21/
Comment on: Rada R. A Case Study of a Retracted Systematic Review on Interactive Health Communication Applications: Impact on Media, Scientists, and Patients. J Med Internet Res. 2005;7(2) p. e18 http://www.jmir.org/2005/2/e18/
Comment in: Eysenbach G. "Is Cybermedicine Killing You?" - Peer Review and Evidence-Based Medicine: Author's Reply. J Med Internet Res. 2005;7(2) p. e39 http://www.jmir.org/2005/2/e39/
J Med Internet Res 2005;7(4):e38)
A recent JMIR article  and corresponding editorial [ ] discuss an error in a review by a Cochrane Collaboration Group [ ]. The articles accurately demonstrate that an error occurred, with the traditional approach to peer review failing.
A solution to this situation is necessary as similar errors could occur in the future. The Cochrane Collaboration attempts to achieve a higher standard than systematic review articles and meta-analysis articles published in other journals. The Cochrane Library claims it is "the best [italics added] single source of reliable evidence about the effects of health care" . Striving for "the best" should include following best practices for peer review.
Rada  advocates extending the Cochrane Collaboration's current practice of open commentary to the prepublication phase. Articles could only be published once there has been extensive commenting by any interested individuals and a consensus has been achieved. Although a good suggestion, there are a few concerns. First, how much time would be necessary before a review period would be deemed appropriate and the article is published? Second, it often can be impossible to reach a consensus among all the reviewers, especially if there were a large number of individuals commenting on a particular topic.
Eysenbach and Kummervold  recommend making it a requirement to invite all primary authors quoted in the systematic review to comment on the review before publication. This suggestion has a lot of merit as this would guarantee that some of the peer reviewers are not only knowledgeable scientists, but also actual experts in the specific topic reviewed.
I suggest taking this a step further. The current Cochrane Collaboration policy is to have 4 peer reviewers for each manuscript . My suggestion is that 2 peer reviewers should be specifically among those whose primary studies have been quoted. One should be from a positive outcome study and the other from a negative outcome study. This would give fair representation to each side on the topic being reviewed. The 2 other peer reviewers could be knowledgeable scientists who are not quoted in the review. These 2 other peer reviewers would be no different than the current standard for a typical journal article that usually has 2 peer reviewers.
Furthermore, each Cochrane review should state the level of the peer review on the title page. For example, a level "A" review would be a review with 4 peer reviews. There would be 1 reviewer from a positive outcome study quoted in the primary review, 1 reviewer from a negative outcome study quoted in the primary review, and 2 additional reviewers who have not been quoted in the review. A level "B1" review would have 4 peer reviews, similar to level "A," but would have only 1 peer reviewer from a primary positive outcome study review the manuscript. A level "B2" review would have 4 peer reviews, similar to level "A," but would have only 1 peer reviewer from a primary negative outcome study review the manuscript. Finally, a level "C" review would have 4 peer reviews but would not have any peer reviewers whose studies were quoted in the primary review.
This approach for peer review may prevent future errors from occurring and maintain Cochrane Collaboration articles as the standard for systematic reviews.
- Rada R. A case study of a retracted systematic review on interactive health communication applications: impact on media, scientists, and patients. J Med Internet Res 2005 Jun 30;7(2):e18 [FREE Full text] [Medline] [CrossRef]
- Eysenbach G, Kummervold PE. "Is cybermedicine killing you?" - the story of a Cochrane disaster. J Med Internet Res 2005 Jun 30;7(2):e21 [FREE Full text] [Medline] [CrossRef]
- Murray E, Burns J, Tai SS, Lai R, Nazareth I. Interactive health communication applications for people with chronic disease. Cochrane Database Syst Rev 2004(4):CD004274. [Medline] [CrossRef]
- The Cochrane Library: an introduction.: The Cochrane Collaboration URL: http://www.cochrane.org/reviews/clibintro.htm [accessed 2005 Jul 1] [WebCite Cache]
Edited by G. Eysenbach; This is a non-peer-reviewed article. submitted 01.07.05; accepted 14.07.05; published 28.07.05
© Joshua Fogel. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 28.7.2005. Except where otherwise noted, articles published in the Journal of Medical Internet Research are distributed under the terms of the Creative Commons Attribution License (http://www.creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited, including full bibliographic details and the URL (see "please cite as" above), and this statement is included.