This study focused on providers of EBP point-of-care summaries, which can be broadly defined as web-based medical compendia specifically designed to deliver predigested, rapidly accessible, comprehensive, periodically updated, and evidence-based information (and possibly also guidance) to clinicians (see Table 1 for definitions). Thus, in order to be included in our analysis, a product had to be an online-delivered tertiary publication (summary) that is regularly updated, claims to provide evidence-based information to physicians and other health professionals, and is to be used at the bedside. As previously stated, the term “point of care” indicates the point where patients and practitioners interact, particularly referring to the context of the provider-patient dyad. Here, “point of care” applies to a summarized reference content describing alternative options in clinical practice, rather than technical solutions optimized for the use at the bedside. We restricted our analysis to summaries published in English as the primary language.

The following online information resources were excluded (see Table 1 for definitions): (1) guideline databases (as they are intended to provide recommendations rather than information); (2) medical meta-lists and search engines, both medicine-specific and general (as they point the user toward the right place to find information rather than providing information themselves) [10]; (3) literature surveillance alerting systems (as they monitor a defined set of journals reporting articles selected for validity and relevance); (4) online books (as they are not regularly updated); (5) original studies reported in medical journals, practice articles, abstracts of papers (ie, primary literature); (6) secondary literature (as it primarily comprises synthesized content, ie, level 2 of the “5S” levels of organization of evidence) [6]. No restrictions were placed on product development status, disease or medical area, or access or charging agreements.

To our knowledge, there is no single repository of online information summaries. In order to retrieve relevant summary products we performed a Medline search using the following terms: (("evidence-based medicine"[Mesh]) AND ("information storage and retrieval"[Mesh])) AND (("online systems"[Mesh]) OR ("point-of-care systems"[Mesh])).

We collected additional information from the references cited in the papers retrieved. Google was extensively used as the search engine to explore products not reported in the medical literature but available on the market. The following terms were used: “medical information system,” “point of care,” and “evidence-based medicine.” We also screened several publisher and librarian association websites, such as the Council of Science Editors [11], the World Association of Medical Editors [12], the Medical Librarian Association [13], the European Association for Health Information and Libraries [14], and the American Medical Informatics Association [15]. Finally, we analyzed the publishing products presented at several scientific information conferences and exhibitions during the period 2006 through 2008, such as the London Online Information Expo and Medical Library Association Meeting and Exhibition.

We repeated our search and collection during the one-year period from January through December, 2008.

For each provider, two reviewers independently retrieved information through an analysis of the official website. As reported in detail below, for any EBP point-of-care summary provider we extracted general characteristics, volume and breadth of the conditions considered, and information regarding the quality of the editorial process and EBP approach to content development (evidence-based methodology). Decisions to select items describing these features were informed by evidence whenever possible. Detailed operational definitions are reported in Multimedia appendix 1.

The features selected were qualitatively described; for editorial and evidence-based methodology indicators, an empirical quantitative evaluation was also included in order to assign a score for each item and rank the EBP point-of-care summary providers. For each quality indicator a point score was assigned: 3 points if the quality indicator was completely fulfilled, 1 point if partially fulfilled or unclear, and 0 points if not fulfilled or not reported. (See Multimedia Appendices 2 and 3.) We arbitrarily decided to award 3 points instead of 2 for adequate fulfilment to give more weight to a more transparent and accountable reporting style and increase the variability within the sample. This policy was somewhat similar to the three-points-for-win rule in soccer [16].

Data were extracted by two independent reviewers (authors RB and LT) who used an ad hoc predefined form. We obtained general features and information on the editorial policy and content development from a thorough analysis of the website pages that were freely available (ie, homepage, about us, editorial policy, and methodology description sections). When subscription to a product was not available at our institution, the free trial and sample topics were used to acquire further information on the content characteristics of the product and the type of output. We assumed that sample topics would likely provide users with the “best” of the product as these parts are often written with the most zeal and attention. When necessary, product editors were contacted by email. When we could not access the content, the products were excluded from the analysis. Disagreements were resolved by discussion between the reviewers and a referee (author LM).

We registered and stored within an electronic archive (December 2008) all the Web pages used to extract data.

Results are presented as median and interquartile ranges to describe the volume and quality indicator scores. The EBP point-of-care products were ranked on the basis of (1) the number of diseases covered (calculated as the number of diseases covered from those in a random sample of ICD-10 chapters); (2) the editorial quality (defined on the basis of adherence to the items reported in Multimedia Appendix 2); and (3) the use of an evidence-based approach (defined on the basis of adherence to the items reported in Multimedia Appendix 3). The relationships between these factors were analyzed by applying the Spearman rank correlation coefficient.

General characteristics and summary content presentation features are summarized in Tables 2and 3. In the EBP point-of-care summaries breadth, we found no variability in the areas of medical conditions covered (data not tabulated). With the exception of Clinical Evidence, all the summary providers reported general information, epidemiology, aetiology, physiopathology, diagnosis, treatment, follow-up and prognosis for each topic but they differed in terms of widening and length. Clinical Evidence focused mainly on treatment alternatives; diagnosis and testing were not systematically covered. Several summary providers presented topics other than medical conditions. For example, 5-Minute Clinical Consult, ACP Pier, DynaMed, eMedicine, EBM Guidelines, First Consult included information about medical procedures; ACP Pier, GP Notebook included, ethical and legal issues; and Dynamed, Harrison’s Practice, Micromedex, and Pepid included drug information with summaries of product characteristics and pharmacokinetic interaction tables. Zynx Health presented complex content and integration with other technologies, such as electronic medical records while drug databases and calculators are distinctive of some other products, such as Micromedex and Pepid according to the shift from summary to systems described in the Haynes model [6].

As of December 2008, we had found 18 EBP point-of-care summary providers. This suggests that several publishing groups and public health organizations are interested in investing time and resources into the development of these products. The overall characteristics of these products tended to vary, and evaluation of their quality is still in its infancy despite the emerging consensus that such information tools are professionally and scientifically essential [4]. Only a few products satisfied our criteria, with none excelling in all. Thus, at present, no clear set of dimensions for deciding among different products can be drawn. The choice of an information tool will depend on the properties of the resource and users’ preferences according to the personal weight attached to different rankings.

One mainstay of evidence-based information mastery is the combination of tools that filter literature for relevance and validity and present summaries easily and in a quickly accessible form at the point of care [3]. Since doctors have significant information needs in their practice [18,19], an important question is whether all these products are reliable and really improve access to high-quality information. While many user-centred or experience/satisfaction analyses have been published [20-25], our evaluation aimed at providing an explicit way to assess the available products other than relying on potentially misleading marketing claims by vendors.

We developed a content validity scale using an evidence-based approach whenever possible. Desirable dimensions were included if there was evidence that not addressing that particular dimension would result in an increased risk of bias. Dimensions were also included where it was clear that information about that dimension was necessary to appraise the reliability of a point-of-care product. For some quality indicators, such as the literature retrieval process and updating, we borrowed our criteria from research on systematic review reporting methods [26,27], assuming that these also would apply to the further synthesized information tools that we included in our review. Other scale dimensions, such as authors’ conflicts of interest and peer review, were based on peer-reviewed medical journals’ policies, extensively debated during the past years [28-30]. Other dimensions that we measured, such as intent to recommend, were included because we judged them to be important, but these were not clearly based on prior research. Only 20% of the products that included recommendations formally graded the strength of the recommendations, whereas doing so is essential to assure transparency and reliability of recommendations [31].

One of the limitations of our study stems from a lack of a clear definition of these products which could have led to a possible selection bias. We set eligibility criteria to select evidence-based summary products defined as portable and comprehensive [1] that Haynes et al would categorize as a summary [6]. Moreover, summary provider eligibility was independently evaluated by two authors. Our study is only a first attempt toward a more comprehensive assessment of this rapidly evolving field. The number of EBP point-of-care summary providers is increasing; in the first months of 2009 at least three vendors, the Journal of the American Medical Association (http://jamaevidence.com), the British Medical Journal (www.bestpractice.bmj.com) and the UK National Health Service (www.evidence.nhs.uk/) launched new point-of-care products. These new products were not included in our survey because they were launched outside the considered time frame.

The major limitation of our study was the arbitrariness of the scoring system. We chose a continuous scale instead of a classical star rating system to allow scores on individual categories to be correlated. Category scores have not been added to make an overall score. Scores allow readers to group EBP point-of-care summary providers according to quality and to detect top performers within categories. Our scoring system can be considered a preliminary approach to rating EBP point-of-care summary providers; other categories could be added.

We did not formally analyze website navigability and usability as this was beyond the scope of our study. Such an analysis might be valuable from the users’ perspective because information on the Web can be communicated in many ways—such as diagrams, animations, and linked pages—which may improve comprehension. These analyses should be carefully interpreted as they suffer from the multiplicity bias as occurs when users are asked to compare known systems with new ones. EBP point-of-care summaries also largely differ according to the comprehensiveness of each topic. Choosing a random sample of ICD-10 chapters as a proxy of the comprehensiveness of a summary may not necessarily be representative of what each provider offers. However, comprehensiveness is a crucial aspect of any information tool when used to answer clinical questions. Further research on the comprehensiveness of these information tools is needed.

None of the associations we postulated turned out to be statistically significant. Thus, on the basis of the criteria we used, editorial quality, evidence-based methodology, and volume appear to be independent. For example, BestBETs scored among the worst on volume (comprehensiveness), with an intermediate score for editorial quality, and the highest score for evidence-based methodology. The search for associations between various desirable factors can be seen as “work in progress,” suggesting that publishers have to balance these aspects, and achieving excellence in all three aspects is difficult.

In the global trend for point-of-care products to inform clinical practice, there is room for improving the quality and increasing the coverage of diseases. Publishers should provide users (or purchasers in general) with transparent, easily accessible, and rigorously determined information regarding editorial processes and content development. Our assessment is intertwined with the quality of reporting. It is possible that publishers favored conciseness of information on their websites and omitted important editorial and methodological details. For instance a publisher may plan to disclose author conflicts of interest, but then does not report this key information on its website, thus diminishing the trustworthiness of its product.

Efforts have been made in the last two decades to improve the quality of reporting of the results of randomized controlled trials and systematic reviews [26,27,32]. However, there is still evidence that methods and reporting can be improved [33-35]. The experience obtained in the field of primary research can be applied to EBP point-of-care information summaries, considering that these point-of-care products are still in the early development. Important initiatives to improve the reporting of health care research, such as the EQUATOR Network [36,37] should also include initiatives to improve point-of-care products.

At present, clinicians who want to select an EBP point-of-care summary to use regularly need to find a balance among several desirable characteristics to inform their choice: according to our criteria, no product appears to be the best. Faced with a choice of summaries, one criterion should prevail. The judgement is complex because in addition to various desirable criteria, many other dimensions could be attractive and drive the choice, such as whether the summary can be used for continuing medical education, contains information addressed to patients, or can be integrated with more sophisticated technologies. Having access to high-quality and well-summarized evidence-based information will not answer all the questions that arise in the doctor-patient relationship, but these summaries help doctors to identify the best options in therapy, diagnosis, or prognosis for their patients. Even the most innovative information system must rely on sound evidence to improve clinical practice; the technology is only the vehicle to make the information accessible. Quality indicators that can be used to evaluate new EBP point-of-care summary providers can be valuable for clinicians, but these can also be useful for librarians, hospital managers, and policy makers who must choose the most appropriate point-of-care summaries to meet their needs.