As this is an update of analyses done in 2008 [11] and 2012 [12], detailed methods and operational definitions can be found in the original publication [11]. Briefly, we defined point-of-care information summaries as “Web-based medical compendia specifically designed to deliver predigested, rapidly accessible, comprehensive, periodically updated, and evidence-based information (and possibly also guidance) to clinicians.” To be included in this review, a product had to be an online-delivered tertiary publication (summary) that is regularly updated, claims to provide evidence-based information to physicians and other professionals, and is intended for use at the bedside. We considered summaries, regardless of their content development status, number of years on the market, clinical focus or specialty, type of access, or charging agreements. We excluded other online information resources such as guideline databases, meta-lists and search engines, literature surveillance alerting systems, online books, and journal articles (ie, primary and secondary literature). Our analysis was limited to products in the English language.

To identify the point-of-care information summaries, we re-examined the eligibility of all products that were included or excluded in the 2008 and 2012 analyses. To find new summaries, we searched Medline from August 2012 to December 2014 with the following terms: ((“Evidence-Based Medicine”[Mesh]) AND (“Information Storage and Retrieval”[Mesh])) AND ((“Online Systems”[Mesh]) OR (“Point-of-Care Systems”[Mesh])). We scanned the references of the papers retrieved and used the Google search engine to identify additional products that may not have been reported in the medical literature. We explored various publisher and librarian association websites (ie, Council of Science Editors, the World Association of Medical Editors, the European Association for Health Information and Libraries, and the American Medical Informatics Association) [13-16], and the 2014 conference proceedings from the Medical Library Association Meeting and Exhibition [17].

One reviewer examined the search results, screened the titles and abstracts of papers identified through Medline, and evaluated the eligibility of products integrating additional information found on product websites. If there was doubt about the inclusion of a product, all authors discussed the eligibility until a consensus was reached. We recorded the reasons for exclusion.

One reviewer extracted information on the general features of each point-of-care information summary. Products that could not be accessed (ie, no subscription available at our institution, no free-trial option, and no response from product representatives to our emails requesting access) were excluded. One reviewer collected data on the general characteristics of products and their content presentation for qualitative (descriptive) evaluation, along with information about the editorial quality, evidence-based methodology, and content volume (breadth of diseases and medical conditions covered) for empirical quantitative analysis. A second reviewer checked the extractions.

For each summary included, we collected the following general details: country of development, year of release, vendor or publisher, marketing claims, format (eg, tablets, mobile devices), access and subscription options, annual costs, and targeted audience. Since the 2012 analysis, we have introduced an additional component: ability to be integrated into an EHR system. This entails the capacity to access information from the point-of-care summary directly through the EHR interface. For example, when a physician clicks on a condition written in the patient record, the physician is directed to a new screen detailing disease information and treatment options. A point-of-care summary search tool may be additionally available on the EHR interface to make free-text and International Classification of Diseases (ICD)-10 code searches.

Content presentation was analyzed in summaries that we accessed. We examined the different outputs (eg, key point summary, paragraphs, question and answers, book chapter-like summary, clinical pathway, clinical scenario), use of formal ontology, flexibility, and reporting of references (with or without general or specific citations). We also assessed products’ adoption of an intent to recommend, use of a formal strength of recommendation system, as well as the availability of continuing medical education programs or credits, other education materials (eg, lessons on statistical analysis or evidence-based methodology), and patient handouts.

Two reviewers extracted information about three key dimensions: quality of the editorial process, quality of the evidence-based approach to content development (ie, evidence-based methodology), and volume or breadth of the medical conditions covered. We described products quantitatively using three separate scores that covered components relevant to each dimension. Disagreements were resolved by discussion between the reviewers. A third author was consulted for any unsolved discordances. All Web pages providing useful data were saved and stored in an electronic archive. When information about a particular component (eg, commercial support or critical appraisal) was unclear or could not be found, we contacted publishers by email requesting additional information and clarification of contents. All emails were stored in an electronic archive.

Volume and quality indicator scores are presented with medians and interquartile ranges (IQR). Point-of-care information summaries were ranked on the basis of (1) editorial quality, (2) the use of an evidence-based approach, and (3) the volume of diseases covered based on a random sample of ICD-10 chapters. Correlations between these three dimensions were assessed by Spearman rank correlation coefficients and their respective P values. Changes in the strength of the products from 2008-2014 were assessed using the matched pairs Wilcoxon signed-rank test. For hypothesis testing, a probability of <.05 was considered statistically significant. All statistical tests were two-sided. Stata software was used for statistical analyses.

General features are summarized in Multimedia Appendix 2. Most of the 26 products were developed by major publishers in the United States (n=12) and United Kingdom (n=8), while others came from the Netherlands (n=4), Finland (n=1), and Australia (n=1). A minority was open access (19%), while most were fee-based (81%) with a median individual subscription price of €244.4 (US$265, £169.52). Regarding their electronic compatibility, over a quarter (7/26, 27%) of products were Web-based only, as others could also be opened on mobile devices. Most products targeted a general audience of health professionals (18/26, 70%), but some were advertised for specific groups such as medical specialists (1/26, 4%), general practitioners (2/26, 8%), nurses (2/26, 8%), emergency medicine doctors (1/26, 4%), pediatricians (1/26, 4%), and rehabilitation professionals (1/26, 4%). Sixteen products out of 26 (62%) could be integrated into EHRs.

Multimedia Appendix 3 presents details of the summary content presentation of the 23 products we could fully evaluate. Products displayed their content in a variety of formats: key point summary, questions and answers, book chapter-like summaries, and clinical pathways (flow charts). Most had a formal ontology for organizing diseases and medical conditions (20/23, 87%) as well as flexible navigation of topic contents (19/23, 83%). Although many products adopted an intent to recommend approach (17/23, 73%), under a third (7/23, 30%) used a formal strength of recommendation system: Grades of Recommendation, Assessment, Development and Evaluation (GRADE) approach [20], the Strength of Recommendation taxonomy (SORT) by the American Academy of Family Physicians [21], or individual systems developed for the product. Just under a half (11/23, 48%) of products awarded continuing medical education credits for searches or featured other programs for continuing medical education. Patient education materials and handouts were available in nearly a third (7/23, 30%) of products, and only a few (4/23, 17%) offered additional educational materials for clinicians such as evidence-based medicine and critical appraisal methodology, lessons on cultural competencies, laboratory manuals, and practice resources.

Figure 2 shows the rank of products based on volume. Disease coverage varied widely: the median volume or coverage of medical conditions was 94% (IQR, 66-100%). The most comprehensive products providing at least one condition per disease category in the four ICD-10 chapters were 5 Minute Consult, Best Practice, Clinical Access, Dynamed, GP Notebook, and UpToDate.

Editorial quality and evidence-based methodology are summarized in Multimedia Appendices 4 and 5; the median scores were 12 (IQR 6-13) and 11 (IQR 4-15), respectively, on a 15-point scale. Five products (ACP Smart Medicine, BMJ Best Practice, Dynamed, Essential Evidence Topics, and UpToDate) received the maximum score for editorial quality. Six (ACP Smart Medicine, BestBets, BMJ Best Practice, Dynamed, EBM Guidelines, and UpToDate) received the maximum score for evidence-based methodology.

The ranking of point-of-care information summaries based on their strength of volume, editorial quality, and evidence-based methodology is shown in Figure 3 (full data reported in Multimedia Appendices 4-6). Best Practice, Dynamed, and UpToDate scored in the highest quartile across all three dimensions. There was a moderate positive correlation between the editorial quality and evidence-based methodology of products (r=.41, P=.0496). No correlations were found between editorial quality and volume (r=.10, P=.64), or between evidence-based methodology and volume (r=.06, P=.80).

Compared to the 2008 evaluation, there were significant improvements in all three dimensions: editorial quality (P=.01), evidence-based methodology (P=.015), and volume (P<.001). Figure 4 shows the evolution of the products in the 2014 assessment that were previously evaluated in 2008.

To evaluate products’ claims to be evidence-based, we adopted editorial policy, content quality, and coverage of medical knowledge as the key indicators of high-quality point-of-care information summaries. In line with the 2008 and 2012 analyses, the purpose of our study was not to pinpoint the “winning” and “losing” products but to assess the maturity of these tools for clinical decision making and encourage transparent reporting of editorial and content development policies by publishers. We further sought to guide readers in the selection of products for individual or institutional use. Since 2008, there have been improvements in the general features of point-of-care information summaries and the descriptions of their editorial approaches, though suboptimal products are still on the market [11].

Several limitations to our study must be noted, including use of editorial policy statements to determine the implementation of a formal and structured peer-review process and the acceptance of commercial support for content development. We acknowledge that there may have been discrepancies between the reporting and actual implementation of editorial policies. Moreover, although we included quality dimensions informed by evidence in our study, our criteria for assessment may be perceived as arbitrary; users of a given point-of-care summary may have different views or experience. Regardless of potential differences in opinions, one observation remains clear: publishers have invested notable energy and resources to raise their quality standards in a limited time. Product maturity and the increasing value of reliable information in medical society may sustain the rising popularity of point-of-care summaries among health professionals.

A particular challenge within our study involved the defining of the intervention and execution of the search strategy to identify relevant interventions for inclusion. Since our first evaluation in 2008, there continues to be a discrepancy in the terminology adopted to describe what we identify as “point of care information summaries”: Web-based medical compendia that are specifically designed to deliver predigested, rapidly accessible, comprehensive, and periodically updated information to health care providers. These products have been additionally referred to as “evidence-based textbooks” [22], “clinical point-of-care tools” [23], navigators, and services [3]. While we recognize that other terms might be used, we have adopted point-of-care information summaries as the preferred terminology, as it embraces several key content elements. Given the rising interest and adoption of these tools, the development of a common term and definition will facilitate their assessment by researchers as well as by hospitals and health care professionals in search of a compatible tool for use. A common definition might also benefit the PubMed MeSH vocabulary. In fact, the MeSH term “point-of-care systems” comprises a broad range of health care technologies outside of our intervention, such as laboratory and diagnostic instruments [24].

The quality of most products is still moderate, which has also been indicated by the few additional surveys evaluating the quality of point-of-care information summaries [22,25-27]. Clinicians should become familiar with the basic concepts that make an information product a credible source of scientific evidence. Health libraries and local knowledge brokers should endorse and give preference to summaries that are committed to policies to improve editorial and methodological rigor, disclose conflicts of interest [28-30], and ensure complete and accessible reporting of the content development procedure. Users should be skeptical about point-of-care summaries that do not transparently describe how information is found (search strategy), selected (cumulative or discretionary approach), evaluated (critical appraisal), prioritized (grading of evidence and recommendations), and regularly updated (literature surveillance) to maintain their relevance to practice. Publishers may be highly skilled in boosting clinical recommendations through propaganda and legally qualified to sell their products to doctors and hospitals. Moreover, the failure to disclose methods for product development is not in the best interests of the medical community, and might, in fact, draw the line between authoritative and fraudulent therapeutic information.

Point-of-care information summaries largely serve high-income countries. However, information on highly effective medicines and interventions are presumably more valuable in low- and middle-income countries. At the same time, in an increasingly competitive market, publishers cannot make the service “free for everyone” because this would affect their sustainability and might facilitate the opportunistic use of these resources. We encourage publishers to align the prices of their products to the purchasing power of a particular country’s physicians through tiered-pricing models and to distribute access through networks active in low- and middle-income countries [31,32]. In addition to their affordability and access, the source of information is critical to the strength and reliability of products.

Dynamed currently has links to over 17,000 guidelines, organized for high or low- to middle-income countries [33]. While the consideration of ready-to-use recommendations is a key first step, more investments in tailoring information to local doctors and other health care providers are needed. For example, information on medicines was never ranked on the basis of the WHO Model List of Essential Medicines, which selects treatments that offer a cure or effective disease management in preference to those that offer only marginal benefit [34]. Doctors are increasingly interested in knowing potential incongruence between investing resources and desired health outcomes [35,36]. In this time of austerity, point-of-care summaries have to do a better job considering the social proven value of medicines.

It is not easy to predict what directions publishers should take to further improve their services. We propose three approaches. First, as summary providers mature and their contents become broader and more complete (eg, information about medicines, recommendations, and guidelines), information must be re-filtered to meet personal practice needs. Users will need to personalize the product, setting filters to isolate specific information (eg, local hospital guidelines) that is relevant to individual clinical practice. This will prioritize information that can engender changes in health professional behavior [37].

Second, high-quality point-of-care summaries should be integrated into computer decision support systems for EHRs. These computer systems may represent the future of clinical decision making in which evidence-based knowledge from point-of-care summaries is linked with patient information from EHRs to generate case-specific guidance messages through rule- or algorithm-based software [3,38]. Computer decision support systems combined with EHRs might be beneficial for the health care provided to patients, although it is hard to demonstrate their association with benefits on outcomes such as mortality [39].

Third, the potential integration of point-of-care summaries into continuing medical education programs should be recognized [40]. Doubts that are raised during clinical consultation can trigger point-of-care searches that provide health professionals with valuable information that can be directly implemented in the visit. Accreditation systems need to recognize the role of point-of-care summaries as an efficient provider of relevant knowledge.

The maturation of point-of-care summaries can be seen as a virtuous circle [41]. It started with an exogenous factor: technological innovation. As health professionals become increasingly familiar with the summaries, their adoption will become self-reinforcing. In a competitive market, this will probably help lower product prices, leading to more potential users. The last 20 years saw the success of PubMed, The Cochrane Library, and, more recently, WikiProject Medicine, which are now integral parts of medical practice. Publishers and developers of point-of-care summaries need to direct their considerable talents and resources to developing strategies to sustain affordable practice and interventions to improve quality of practice. This change of focus can support their development as indispensable professional tools.