For a specific clinical question, a well-conducted systematic review is considered the gold standard for the identification of all relevant articles [31-33], since it implies a systematic and comprehensive search, as well as an appraisal of the articles’ scientific rigor. Moreover, recent systematic reviews address questions of clinical interest for which relative uncertainty exists and are thus a good proxy of the questions that may arise in real clinical practice.

We selected 30 clinical questions from a wide spectrum of topics of general interest, on the basis of 30 recent Cochrane systematic reviews (Table 1, Multimedia Appendix 1). We favored Cochrane reviews because of their well-established quality and methodological rigor [34-36], and focused on reviews that included randomized controlled trials, as these constitute the highest level of evidence for questions about therapeutic interventions. In February 2010, we searched the Cochrane Database of Systematic Reviews [37] for all reviews on therapeutic interventions published since the beginning of 2010. To focus on the most recent reviews, or reviews that were recently updated, we restricted the search to citations with the following record status: new review, new search, or conclusion changed. Of the 287 potentially eligible reviews, we excluded those with fewer than 4 studies (n = 113), those that were less relevant for a general audience (n = 82), and those with composite questions that would result in complex queries for clinical practice (n = 62). All clinical trials included in the remaining 30 reviews that were also retrievable in PubMed defined the subset of relevant studies used in our assessment.

We selected search terms using a predefined procedure that we determined before applying the searches strategies. Following the PICO framework (Figure 1), we categorized the exact wording of the objective and selection criteria of each review’s abstract into four sections for the population, interventions, comparison, and outcomes [22,24]. When we could find no information in the abstract for one of these categories, we retrieved the relevant keywords from the methods section (eg, type of intervention or type of outcome). Then for each PICO category, we extracted the smallest set of search terms that best expressed the clinical question. This predefined procedure was performed by 2 assessors (TA and AM), both trained in clinical epidemiology and evidence-based medicine. In a learning phase, together they extracted search terms from a pilot sample of 10 systematic reviews. Then they independently extracted PICO search terms from the 30 reviews of the study sample and agreed on the smallest set of terms that best expressed the clinical questions. They resolved discrepancies by consensus. Finally, all coauthors, including experienced clinicians, approved the retained search terms. For half of the reviews, PICO’s comparison component shared common terms with interventions (eg, composite vs single intervention), and in these cases no term was retained for comparison. Examples of search term extraction are shown in Multimedia Appendix 2.

To obtain the final search query (Figure 1), search terms were combined with the Boolean operator OR within each PICO category, so as to increase sensitivity. Then all categories were combined with AND to retrieve only the citations matching all PICO elements [22-24]. For example, for a review on the effects of oral mucolytic agents in adults with stable chronic bronchitis or chronic obstructive pulmonary disease (COPD) [38], the final full PICO query was ((chronic bronchitis) OR COPD) AND mucolytics AND placebo AND exacerbations). Search terms consisting of groups of words (eg, chronic bronchitis) were put into parentheses, not in quotes. The detailed wording of all 30 queries can be found in Multimedia Appendix 3.

The purpose and context of the search determine the choice of search strategy. Our objective was to identify search components and tools that could help clinicians build more effective strategies to answer questions at the point of care. Therefore, selecting sophisticated strategies used for performing systematic reviews [39], such as the Cochrane Highly Sensitive Search Strategy [40], would have been inappropriate. Since there is very little evidence on strategies adapted to our purpose, we devised our 15 search strategies (Table 2, Figure 1) based on the few expert recommendations available [22-24,26,27,29-31]. We favored search components and search tools that are applicable and easy to use in clinical practice.

We thus obtained strategies by varying the following parameters. First, the search query was either the full PICO query or a truncated PIC query that did not include terms of outcomes. Although the full PICO is usually recommended for clinicians to help them formulate more precise searches [22-24], such strategies may also miss relevant articles [40], as the outcomes are less often mentioned in the abstract or assigned Medical Subject Headings (MeSH) in PubMed. Moreover, formulating shorter PIC queries may be less burdensome in practice and may be more appropriate for clinicians who are interested in many potential outcomes. We entered search terms using PubMed’s automatic term mapping, which systematically searches each term in the MeSH thesaurus, as well as the “[all fields]” tag (Multimedia Appendix 2). We did not test strategies using the restriction to MeSH terms, as few clinicians manage to perform the necessary steps for this restriction in practice [21], and this proportion hardly increases after medical residents take a search tutorial [29].

Second, we combined these queries with either the broad or narrow filter of the Clinical Queries for therapeutic interventions. We chose Clinical Queries over other filters meant to improve the retrieval of high-quality randomized controlled trials [39] because Clinical Queries were specifically designed to help clinicians answer their questions on therapeutic interventions and are implemented in PubMed [27]. To use these filters, clinicians simply need to enter their search terms on PubMed’s Clinical Queries page [26]. After taking a search tutorial, about 80% of clinicians adopted the use of Clinical Queries [29]. Third, we repeated these strategies adding search limits [25] that restricted the searches to citations in English and research on humans. PICO searches with human and English limits were repeated with a further restriction to a set of 119 core medical journals listed in the Abridged Index Medicus (AIM) [28]. We did not further add the limit “clinical trials [pt],” as it would have been redundant with the Clinical Queries filters.

Finally, the last 3 searches tested the PubMed link to related articles, a search tool that identifies content similarity in the title, abstract, and index terms [41]. To use this tool in practice, clinicians must identify a citation they consider to be potentially relevant in a first search output (ie, a citation that best corresponds to their PICO question) and click on its link to related articles. They can then scan a new search output with citations ranked by content similarity with the initial citation. To test this strategy, 2 assessors (TA and AM) screened the first page of search #11 (PICO, therapy narrow filter, limited to human studies in English), a search strategy that yielded short outputs (Table 2), and identified by consensus the 3 citations that were closest to the corresponding PICO question. They performed this selection being blinded to the citations included in the review, so as to correspond to the situation that clinicians encounter in practice. Search performance was assessed on each output of these 3 citations’ related articles. We performed this strategy based on 3 citations, instead of 1, to minimize the impact of the subjective component of the choice of initial citation.

We applied all 15 strategies for each of the 30 clinical questions (450 searches in total). We restricted each search to the date when the corresponding review was assessed as up-to-date, so that the time frame of the search was the same as its corresponding review. For each search we collected the number of articles in the output, the number of gold standard articles retrieved, and their position in the output. Then we computed the sensitivity (also called recall: the proportion of relevant papers that were retrieved) of each search and its positive predictive value ([PPV], also called precision: the proportion of the output corresponding to relevant papers), as defined in Figure 2 [27,31,33].

Because most physicians screen only 2 pages of the PubMed output, or at most 40 items [19,21], our primary outcomes for search performance were sensitivity and PPV for this cut-off. However, we also analyzed overall sensitivity and PPV on the full search output and further explored how these properties varied according to the cumulative number of items screened in the output.

We summarized the performance of the search strategies over the 30 clinical questions using nonparametric statistics and box plots. Filtered versus unfiltered searches were compared using the Wilcoxon signed rank tests (paired). Finally, we explored whether retrieval performances differed according to the characteristics of the corresponding review. All analyses were performed using R 2.12.1 software [42].

The 30 systematic reviews addressed a broad pattern of clinical topics and were produced by 15 different Cochrane groups, with a range of 1–4 reviews per group. Of these, 13 reviews (43%) had their record status labeled as new review, and 13 others (43%) as new search with no change to the conclusion and 4 (13%) with conclusion changed. They included a median of 15 studies (interquartile range [IQR] 7–23, range 5–49), of which 85% (IQR 66%–100%) were retrievable in PubMed (Table 1). This led to a median of 12 relevant clinical trials per review (IQR 7–18) that we considered to be the gold standard for each clinical question.

The predefined extraction procedure of search terms from the reviews’ abstracts led to a median of 2 terms (IQR 1–2) for population 1 term (IQR 1–2) for interventions, 1 term (IQR 0–1) for comparison, and 2.5 search terms (IQR 2–3) for outcomes. The required number of search terms was variable, as these terms were strictly tailored to the reviews’ questions. Indeed, these were sometimes more complex or consisted of grouping of words (eg, generalized onset tonic–clonic seizures). Overall, queries resulted in a median of 4 terms (IQR 3–6) for the PIC query and 7 terms (IQR 5–8) for the PICO query (for detailed wording, see Multimedia Appendix 3).

We observed important differences in the sizes of the output across search strategies (Table 2). Unfiltered PIC queries resulted in the largest outputs, with a median of 173 items, while the PICO queries halved the output size. The use of the Clinical Queries broad filter reduced the output by about 20%, and use of the narrow filter reduced it by about 80%. In contrast, searches based on related articles typically retrieved hundreds of articles.

The sensitivity and the positive predictive value (PPV) were also highly variable within each search strategy (Figure 3). When the full outputs were screened for relevant studies, about 85% were detected by PIC queries and 69% by PICO queries (Figure 3A). Overall sensitivity remained comparable when we used Clinical Queries filters (with or without limits), although the Clinical Queries narrow filter was associated with slightly lower overall sensitivities (Table 1A). In contrast, the use of the AIM limit systematically lowered sensitivity to about 15% (Figure 3A). The overall sensitivity of searches based on related articles was also extremely variable, with a median of about 60%.

When the screening of relevant articles was limited to the first 2 pages (ie, 40 articles), sensitivity dropped to 10% for unfiltered searches and PPV decreased to 2.5%–10% (Figure 3B). In contrast, all search strategies that used the Clinical Queries narrow filter (S4, S5, S10, and S11 in Table 2) showed significantly higher sensitivities, overall around 50% (P < .001 when compared with their corresponding unfiltered queries) and higher PPV values, between 20% and 30% (P < .001) within the first 2 pages of the output. When looking at the 30 questions individually, adding a Clinical Queries narrow filter to a query increased the sensitivity of the search for 21 questions (70%), kept it stable for 6 (20%), and decreased it for 3 (10%), whereas PPV increased in 27 questions (90%) (Table 1B). Moreover, adding a Clinical Queries narrow filter to a PIC query reduced the risk of finding no relevant articles in the first 2 pages from 11 questions (37%) to 4 questions (13%). The broad Clinical Queries filter did not improve search performance; nor did searches using related articles and the additional use of limits. Overall, PICO queries had slightly higher performances than PIC queries, although the differences were not statistically significant.

The exploration of search performance according to the number of articles screened in the output showed that sensitivity rose gradually up to about 100 items in unfiltered searches (Figure 4). In contrast, the sensitivity of searches using the Clinical Queries narrow filter rose much more steeply, peaking earlier at 50 to 60 articles screened. Finally, we found no significant association between search performance and characteristics of the reviews, namely the absolute number of relevant articles included, the number of PICO search terms required to summarize its question, or the presence of mortality as an outcome (data not shown).

The main limitation of our study is that we assessed search strategies based solely on their retrieval performance. The searches were not performed by clinicians at the point of care, so we did not capture the iterative process of searching PubMed, based on trial and error [43,44], and could not assess whether relevant articles would be identified in practice. Furthermore, we did not examine the impact of search strategies on medical decisions and patients’ outcomes. However, performing searches that retrieve relevant articles is a prerequisite for their further use in clinical practice. Moreover, by using predefined search procedures, we were able to isolate and compare the effect of several components of search strategies in relation to a specific clinical question. As we focused on retrieval, we considered each relevant article to be equally important, although this may not be the case. In particular, we did not examine selection bias attributable to suboptimal identification of relevant articles. Finally, our results apply only to therapeutic interventions, assessed by clinical trials, and cannot be generalized to questions on prevention or diagnosis [32]. For the latter, similar studies could be conducted using alternative Clinical Queries filters designed to improve the retrieval of high-quality studies on diagnosis or prognosis [26].

Among its strengths, our design allowed an unbiased comparison of strategies, by controlling important sources of variation such as clinicians’ searching skills or previous knowledge [29,30]. This exploration was broader and more systematic than in previous studies. Moreover, the tested search strategies relied on search components that are easily applicable in practice, as they are directly implemented in PubMed. Finally, we used reputable systematic reviews to identify relevant studies on specific clinical topics.

Countless PubMed searches are performed daily by clinicians, but the effectiveness of this activity is poorly understood [20]. We have shown that a well-formulated PIC query used with the Clinical Queries narrow filter was most likely to retrieve relevant clinical trials within readable outputs. These results can help clinicians build more effective strategies to answer their questions at the point of care, and thus reduce the gap between evidence from clinical trials and its actual implementation in practice. Further research should focus on the performance and clinical usefulness of selected strategies when they are performed in real practice [45]. Meanwhile, PubMed remains a perfectible tool. Areas of improvement include the development of content filters for specific clinical disciplines [46], or the implementation of new search interfaces that help clinicians formulate effective queries [24] and conduct parallel searches combining methodological and content filters at will.