In October 2011, we performed Google queries searching for the phrase “list of medical wikis” translated in the 10 most spoken languages on the Internet (English, Chinese, Spanish, Japanese, French, Portuguese, German, Arabic, Russian, and Korean), using the Google translation tool when necessary [23]. The phrase was expanded as far as possible within the limit of 500 resulting pages. The English query was filtered in order to remove an extensively cited page, which has been kept once for data extraction [24]. Every resulting page was browsed in order to extract Internet addresses (uniform resource locators [URLs]) linking to potentially relevant sites (Multimedia Appendix 1).

Second, we searched PubMed and Web of Science (using “wiki” AND [“medic*” OR “clinic*”]) and Literatura Latino-Americana e do Caribe em Ciências da Saúde (LILACS) (using “wiki”) in full texts for articles published until September 2012. Every open-access abstract and open access article was read, coupled with Web searches when necessary, in order to identify any potentially relevant URL (Multimedia Appendix 2).

Finally, we included any other potentially relevant URL retrieved through Web extra-browsing or expert advice, until September 2012. One author (AB) made all data extractions of the screening.

Websites were included if they were (1) accessible from a public Internet protocol address; (2) operating a wiki tool, defining a “wiki” as “a type of content managing system (CMS) used for collaborative edition, where the content is created without any defined owner” [6], excluding wiki-based platforms used as non-collaborative CMS, like Wikinu [25], and websites where a collaborative edition was allowed on owned contents, like Google Knols [26]; (3) aimed at building some knowledge relevant for a clinical practice, defining “clinical” as “of or relating to the bedside of a patient, the course of a disease, or the observation and treatment of patients directly” [27], excluding medical topics not directly linked to the care of patients (medical research, medical informatics, biomedical sciences, medical curriculums, pharmacology, public health), and topics not specifically interesting physicians (other health care disciplines, patient information, first aid); and (4) explicitly targeting physicians or medical students in audiences. Wikis orientated toward general public, like Wikipedia, were excluded [28]. In addition, websites were excluded if they were dysfunctional, explicitly interrupted, only aiming at displaying external resources. Some clinical-oriented wikis, like Medical Matters Wiki, were excluded as bibliographic resources [29].

The inclusion and exclusion was done by 2 authors (AB and LL), and disagreements were solved by discussion.

The Google search yielded 341 pages, including 27 linking to some potentially relevant URLs. After extraction and removing duplicates, 141 URLs were collected (Multimedia Appendix 1). The literature search yielded 133 articles, 104 after removing duplicates. After identification of potentially relevant URLs and removing duplicates, 38 URLs were collected (Multimedia Appendix 2). Four additional potentially relevant URLs were retrieved from other sources. Merging all results and removing duplicates, 176 potentially relevant URLs were finally collected (Figure 1, Multimedia Appendix 3).

Of the 176 collected URLs, 31 met the inclusion criteria. Six of them became inoperative during the study. Finally, 25 wikis were retained for analysis [35-59] (Figure 1; Multimedia Appendix 3).

From this international review, we identified 25 medical wikis dedicated to clinical practices. The majority were in English and four were bilingual. They had various purposes, dominated by encyclopedic perspectives (44%), and most were specialized (64%). The MediaWiki software was commonly used (68%), often in its native form (60%). Site owners were mostly non-profit organizations (40%) and individuals (24%); only two were universities. While practicing physicians were major contributors (88%), medical learners (36%) and lay persons (16%) sometimes contributed.

Cross-reading our results, the relevancy for clinicians of the medical wikis can be discussed according to four information properties: accuracy, readability, reliability, and currency. Accuracy may be impaired in wikis not displaying a review policy (60%) and in those not delivering rules for organizing content (80%) [63,64]. The articles from encyclopedic wikis presented characteristics less relevant for professional use than the others, including more pictures, videos, and external resources but fewer posology details. The Flesch reading ease scores were globally low, especially for encyclopedic articles. In regard to reliability, 64% of wikis fulfilled less than half of the information framework quality criteria. In addition, articles were poorly referenced, and evidence level notifications were exceptional. Finally, 88% of the wikis had fewer than 50% of articles revised in the last year, and 24% of the sites were almost unused.

Our review may not have been exhaustive as the Google search was restricted to lists of medical wikis and several sites reported in the health literature were not accessible. Furthermore, the Web 2.0 field is rapidly changing, and some new medical wikis may have emerged since October 2012. Re-browsing the lists of medical wikis used in this study, we found only one relevant wiki after the inclusion period: the Australian Cancer Guidelines Wiki [65]. Among the 25 included sites, Medpedia and WardWiki have been discontinued [35,47], and a few changes occurred in the structure of the others: Open Anesthesia has been reorganized [49], WikiEcho and MedRevise changed their “skin” [42,51], and Oncologik added a missing link to its owner [54].

Among the tools available for assessing the quality of health information on websites, none is currently validated and none is fitted either to wikis or to a professional audience [66,67]. The HSWG IQ tool does not take into account collaborative features, as acknowledged by Dobrogowska-Schlebusch [15], and it has been removed from the Web [19]; the DISCERN tool targets health consumers and is restricted to information on treatments [68]; and the Bomba and Land Index has also been designed for health consumers [69]. Numerous items are common between these questionnaires and major guidelines such as the eHealth code of ethics [70], the American Medical Association guideline [71], or the eEurope 2002 quality criteria [72]. The HONcode ethical code of conduct is unique to provide specifications for collaborative websites [31,73]. For example, the item “is the information referenced?” will be transposed for collaborative websites as “is there a statement asking platform users to give references to the information they provide?”. Such specifications do not directly apply to the content, but indirectly through the editorial framework. However, the right influence of the framework on the content deserves to be investigated in future research projects.

The relevancy of low readability scores, corresponding to college and higher, is arguable since medical doctors have de facto a high level of reading. It has been long demonstrated that readability impacts both the understanding and the cross-reading ability, even for highly educated readers [74], and the need for simplicity is expressed by clinicians themselves for practice guidelines [4]. The relevancy of the Flesch reading ease test for medical writings is also debatable, but more specific tools are not yet validated [75]. Although it includes adjustment parameters adapted to several languages [34], a linguistic bias cannot be excluded in this study since multilingual comparisons have not been documented.

To check the validity of the estimation of annual editorial activities using Google, we measured the agreement between the number of content pages declared on the site and the corresponding estimate from the Google search engine, for 20 wikis. Although there was a strong agreement (Spearman correlation coefficient=.88, P<.001), automated page creation and vandalism may bias both figures.

Our results suggest that no medical wiki meets all four information properties needed by clinicians. The encyclopedic format does not seem to fit in terms of both accuracy and readability. However, whatever the wikis’ purposes, the organization of contents is often unclear, apart from very focused purposes such as oncology protocols, where the knowledge granularity is adapted to a particular audience [54]. The Medical Subject Headings (MeSH) indexing system is sometimes integrated, but it requires specific training for contributors, which is challenging in a multi-authoring context [76]. Whereas some semantic utilities can help manage indexation constraints [10,77], add-ons aimed at improving either medical knowledge management or ergonomics are rarely implemented in medical wikis. If such gaps impact both accuracy and readability, they may also hamper the involvement of users. Contrary to pure knowledge content, the frequent clinical case reports in medical wikis, supporting the emergence of concrete questions of practice, are likely to meet strong clinical interest.

Reliability is widely, and sometimes critically, impaired by lack of management. Although authoring transparency requires both technical and policy supports [5], our framework assessment particularly shows gaps in users’ disclosures and editorial policies. Since almost only purpose-built platforms are able to manage detailed user data, technical issues are important. Among open communities, only 48% of medical wikis ask for credentials to register, with two requiring some proof [35,36]. As an alternative, users’ medical skills can be assessed during an automated registration including medical tests [58,78]. Interestingly, the fully opened Wikipedia’s articles are commonly consulted by clinicians and medical students [79], while their relevancy has been recurrently questioned [7,14,16-18,21]. However, Wikipedia, including its Wikiproject Medicine, cannot respond to specific clinical needs as it does not target any specific audience [28]. As an encyclopedic media, it is also likely to meet the limitations highlighted in this study.

In most wikis, weak and poorly collaborative activity jeopardizes content updates. The talk pages, when available, are exceptionally used, and the discussion threads included in forums or social networks are not directly connected to content pages [80]. As a consequence, adversarial debates are lacking, although they are a foundation for building evidence [3].

Users’ regulation in wikis is complex since the lower the control of their editors, the higher their growth [81]. For example, Wikipedia’s English article on atrial fibrillation has been revised approximately 1345 times and discussed 150 times [82], and the article on the recent drug dabigatran 555 times and 35 times respectively [83]. Apart from the severe reliability issues due to anonymity in Wikipedia [84], it has been shown that its development, based only on volunteering, leads articles to be unevenly readable, complete, and reliable [17,20,85]. In our study, we paradoxically observed the highest page revision and discussion levels in a small wiki reserved to a closed community [54]. This finding suggests that a strong user commitment can overcome volunteering limitations.

Although multi-authoring requires a thorough organization [86], communities attached to medical wikis are often poorly structured. Super-user nominations are usually opaque, and only one wiki provides a standardized peer-review process [39]. As implemented in two wikis, the extent of users’ rights can depend on their participation level [35,58], which represents a reward for authors [87]. However, in order to open scientific debates, the organization of bottom-up relations between users should be further considered [88]. In this way, the public expertise promoted by Wikipedia, which is based on consensus, uses a complex and democratic moderation system, detailed editorial rules, and standardized peer-reviews [21].

While the HONcode principle about the authoritativeness of the information protects the moderators’ privacy by allowing their anonymity [31], it cannot guarantee the trustworthiness of what they have written [84,89]. The professional scope of our review highlights a lack of audience specifications in health information quality initiatives, in particular for collaborative applications where readers and writers are mixed altogether. The extensive review of social media by Grajales et al provides a useful tutorial for health care professional end users, which may be a first step to building more detailed guidelines for professional health information on the Internet [7]. Indeed, some professional knowledge may generate adverse outcomes, as information on drugs with potential for misuse is commonly sought on the Internet [90]. Therefore, as included in the wikiproject Medicine of Wikipedia [21], a policy specifying the nature and the limits of publicly accessible content is critical, and a model for displaying health information is needed [67,73].

Among eight medical wikis including learners’ contributions, five include spontaneous undergraduate or postgraduate students’ contributions. The three others have a formal educational goal, targeting postgraduate students or practicing physicians in continuous medical education [49,58,59]. Educational goals may represent an alternative to mere volunteering since learners’ contributions can be part of their curricula. As works performed in training are frequently based on clinical cases as starting points for gathering scientific evidence [91,92], the wiki principle seems particularly fitted to archive, share, discuss, and gradually improve the related materials [93]. From a theoretical point of view, the wiki medium, as an asynchronous communication tool, embodies learning principles based on constructivism and cooperation [94]. Nevertheless, if Internet-based educational programs can be an alternative to live interactive workshops [95], the effectiveness of collaborative writing applications in medical education requires further research [12,14].

The 25 medical wikis reviewed present various limitations in their format, management, and collaborative features. Encyclopedic wikis have less accurate and readable content. Reliability is widely impaired by lack of transparency. Currency is commonly jeopardized by low editorial activity. Professional medical wikis may be improved by using clinical cases, developing more detailed transparency and editorial policies, and involving postgraduate and continuing medical education learners.