We first selected the top 6 journals in the “Medicine. General & Internal” Journal Citation Reports category, according to the impact factors computed by Clarivate [23,24]. Next, we selected all the PubMed-indexed SRs published in the 6 journals between 2012 and 2021 and for which the free full text was available on PubMed Central. The time period was chosen arbitrarily, with the objective of obtaining at least 60 SRs. The following PubMed query was used: ‘(“The New England Journal of Medicine”[Journal] OR “Lancet London England”[Journal] OR “JAMA”[Journal] OR “Nature Reviews Disease Primers”[Journal] OR “BMJ Clinical Research Ed”[Journal] OR “Annals of Internal Medicine”[Journal]) AND “loattrfree full text”[Filter] AND 2012/01/01:2021/12/31[Date - Publication] AND systematic review[Filter]’.

The exclusion criteria were as follows: (1) articles other than an SR, (2) the absence of a published search query, (3) the use of queries in multiple parts that had to be assembled, (4) the absence of a query specifically built for PubMed, (5) a query that did not return any results, (6) a query that returned more than 100,000 results, and (7) a query with only MeSH terms or without MeSH terms. The sorting was carried out by a single researcher (VL).

The query was extracted from each included SR and inserted into the PubMed search bar. PubMed has a feature called automatic term mapping (ATM) [25]; when terms not enclosed in quotation marks are inserted in the search bar, they are automatically transformed into a query segment that contains several descriptors, such as [MeSH terms], [tiab], and [all fields]. To ensure greater reproducibility, we checked for the automatic transformation of queries. This step was important because PubMed’s ATM feature might add MeSH terms to query initially considered to be free of such terms. Hence, we always retrieved the query formatted by PubMed’s ATM (henceforth referred to as V1; Figure 1).

For each SR, V1 was transformed into a V2 query by replacing each MeSH term in the query with a free-text term that had to be present in the title or in the abstract. To do this, we simply replaced the [MeSH] tag with a [Title/Abstract] tag. Hence, the resulting V2 did not contain any explicit [MeSH] tags (Figure 1). Lastly, the V3 (MeSH-only) query was obtained by transforming all free-text terms into MeSH terms. It should be noted that terms stated as MeSH terms in the query but that do not actually exist in the MeSH thesaurus are ignored by the PubMed engine; this is equivalent to deleting the terms (Figure 1 [26]).

The transformations from V1 to V2 and V3 were the same for all queries, regardless of whether they contained MeSH terms only or free-text terms only. However, we noted that some PubMed filters are based on MeSH terms [27]. It would therefore not be relevant to convert these terms into free-text terms. We drew up a list of these terms so that they were not transformed and were still able to serve as filters. Those 14 terms are “80 and over,” “adolescent,” “adult,” “aged,” “animals,” “child,” “female,” “humans,” “infant,” “male,” “middle aged,” “newborn,” “preschool,” and “young adult.”

Hence, each SR had a query written by the SR’s authors (a combination of MeSH and free-text terms; V1), a free-text-only query (V2), and a MeSH-only query (V3). Therefore, we intend to interpret the comparison of V2 with V1 as the added value of MeSH terms, and we intend to interpret the comparison of V3 with V1 as the added value of free-text terms.

Each query was submitted to the PubMed search engine, and the results were retrieved and sorted by the “Best Match” option. If there were more than 10,000 results, only the first 10,000 results were retained; in fact, PubMed does not allow more than 10,000 results to be extracted. The results were identified by their PubMed Identifier (PMID).

For each SR, the “gold standard” (GS) consisted of the articles selected by the authors of the SR. Each SR was read in order to extract the list of PMIDs selected by the authors. This work was done “by hand” by 4 researchers (VL, RB, BLG, and AH). Publications cited in the SR but not indexed in MEDLINE were not analyzed. If the reference section did not contain the items selected in the SR, data extraction from supplementary files allowed for the completion of the GS.

For each SR, we obtained 4 lists of PMIDs: the GS, those retrieved by V1 (MeSH and free-text terms), those retrieved by V2 (free-text terms only), and those retrieved by V3 (MeSH terms only). For each list, we computed the sensitivity (also referred to as “recall”) and the positive predictive value (PPV; also referred to as “precision”) with respect to the GS. We then computed the F₁-score, which is the harmonic mean of the sensitivity and the PPV.

For each query i (V1, V2, and V3), the odds for the PPV was defined as the ratio between 2 numbers:

Next, for a given SR and using the same GS, the odds ratio (OR) of query₂ to query₁ for the PPV was defined as:

Likewise, the odds for the sensitivity of each query i (V1, V2, and V3) was defined as the ratio between 2 numbers:

Hence, for a given SR and using the same GS, the OR for query₂ versus query₁ with regard to sensitivity was:

We computed the respective ORs for V2 versus V1 and V3 versus V1 for the PPV and the sensitivity:

An OR of 1 means that the queries have the same level of performance with regard to the chosen indicator. An OR<1 denotes worse performance, and an OR>1 denotes better performance.

Qualitative variables, binary variables, or discrete variables with very few modalities were expressed as the frequency (percentage). Quantitative variables were expressed as the mean (SD) when symmetrically distributed and the median (IQR) when not. The independence of 2 qualitative variables was probed in a chi-square test.

All statistical tests were 2-sided. The threshold for statistical significance was set to P<.05. The 95% CI of a proportion was calculated using the Wald method. Statistical analyses were performed with R software (R Core Team), RStudio software (Posit PBC), and the R metafor package [28-30].

The research was performed using publicly available documents. It did not involve individuals or personal data. Approval by an institutional review board was not required.

The SRs used to compile the set of queries were selected by a single researcher (VL; Figure 2).

A total of 59 SRs were selected for analysis, which contained both MeSH terms and free-text terms (Table 1 and Multimedia Appendices 1 and 2 [26,31-88]).

Of the 59 selected SRs, 29 (49%) came from The BMJ, 19 (32%) came from the Annals of Internal Medicine, 6 (10%) came from The Lancet, and 5 (9%) came from the Journal of the American Medical Association. The publication dates were evenly distributed; the mean publication year and the median publication year were both 2016.

The countries of origin of the first authors were available for 49 (83%) SRs. The 3 most frequent countries of origin were the United States (21/49, 43%), the United Kingdom (5/49, 10%), and Canada (5/49, 10%).

The queries contained a median (IQR) of 43 (17.0-98) terms. The median (IQR) number of MeSH terms in the V1 queries was 6.0 (3.0-19.5). The median (IQR) proportion of MeSH terms relative to all terms in queries was 18.5% (13.7-25.5).

The V1 queries returned a total of 206,095 items, of which 1628 (0.79%) were included in the GS (Table 1). The V2 queries returned a total of 157,698 items, of which 1473 (0.93%) were included in the GS. In other words, an average of 820.29 additional articles per SR had to be screened for V1, relative to V2. Furthermore, V1 retrieved an average of 2.62 additional relevant articles, when compared with V2.

The median (Q1-Q3) sensitivities of queries V1 and V2 were 77.8% (62.1%-95.2%) and 71.4% (42.6%-90%), respectively (Table 2). The median (Q1-Q3) PPV of queries V1 and V2 were 0.9% (0.3%-2.8%) and 1.1% (0.3%-3.4%), respectively. The median (Q1-Q3) F₁-scores of queries V1 and V2 were 1.8% (0.7%-5.4%) and 2.2% (0.7%-6.1%), respectively. A graphic visualization of the sensitivity and PPV per SR showed that the addition of MeSH terms to a query typically increased the sensitivity but decreased the PPV (Figure 3). Furthermore, it can be seen that the transition from V2 to V1 had no effect for many SRs.

Overall, V1 provided 8.49% more of the GS’s items than V2 and 35.55% more of the GS’s items than V3. V2 provided 27.06% more of the GS’s items than V3. The ratio between the number of GS references retrieved by V1 and the number retrieved by V2 was within the interval (0-1.05) in 66% (39/59) cases (Figure 4). In 59% (35/59) of cases, the ratio was 1 or less. In other words, the transition from V1 to V2 did not have a marked effect on the number of relevant articles retrieved for more than half of the SRs.

We also calculated the ORs for the number of relevant articles retrieved by V2 relative to V1 (Figure 5 [26,31-88]). Overall, the OR (95% CI) for V2 versus V1 was 0.55 (0.38-0.78) for sensitivity and 1.26 (1.03-1.54) for the PPV (Figure 5). The OR (95% CI) for V3 versus V1 was 0.31 (0.23-0.41) for sensitivity and 3.11 (2.15-4.48) for the PPV (Multimedia Appendix 3 [26,31-88]).

The objective of this work was to quantify the utility of MeSH terms in SR queries. To this end, we retrieved the queries drafted by the authors of 59 SRs published in 4 prestigious medical journals. We then modified the V1 query to give a free-text terms only query and a MeSH-only query. Finally, we calculated the 3 queries’ sensitivities, PPVs, and F₁-scores.

Our first key observation was that MeSH terms typically accounted for a nonnegligible proportion (on average, 20.4%) of the terms in the query. Second, the removal of MeSH terms from SR queries decreased the sensitivity (by 6.4%, on the median) and increased the PPV (by 0.2%, on the median). In other words, queries containing both MeSH terms and free-text terms yield an average of 2.62 additional relevant papers per SR, necessitating the screening of an additional 820.29 papers. The cost of screening an additional collected paper was therefore 313.09, which was slightly more than triple the mean reading cost associated with free-text terms only queries (88.67). Third, our results indicated that the deletion of MeSH terms had no effect on the number of relevant articles retrieved for 35 (59%) of the 59 reviews.

The results of a previous study were similar to those found here; 95% of the relevant articles were retrieved in 67% (49/73) of the analyzed SRs when the query contained free-text terms alone (relative to the V1 query with a mixture of MeSH terms and free-text terms) [22]. Another study with a similar objective gave significantly different results; the free-text terms–only query was 25% less sensitive than MeSH-only query [15]. However, it should be noted that (1) the latter findings were based on a single query, and (2) the MeSH terms were converted to free-text terms manually, with a relatively limited set of synonyms used in the free-text terms strategy.

Furthermore, 3 messages should be highlighted. First, MeSH terms remain an indispensable tool for SRs despite the significant advancements in free-text search engines, especially in an era where the quality of SRs is declining [89]. Second, free-text terms appear to contribute more effectively to the retrieval of relevant articles compared with MeSH terms. Third, mixed queries (combining free-text and MeSH terms) exhibit poor PPV; for rapid literature reviews, it is preferable to use either MeSH terms or free-text terms exclusively.

Our study involved queries developed by experienced researchers; choosing free-text terms can be challenging and requires expertise. It is possible that clinicians with limited experience in literature searching struggle to choose free-text terms effectively, and yet, bibliographic research among clinicians is essential [90]. MeSH terms offer a distinct advantage over free-text terms by covering a broad range of vocabulary, which can be particularly beneficial for clinicians, early-career researchers, or nonnative English speakers. In such cases, incorporating MeSH terms can help clinicians construct more comprehensive and effective queries.

The GS comprised solely MEDLINE-indexed documents with a PMID. This choice was restrictive but technically essential, given that the 3 queries were submitted to the PubMed search engine. However, our restriction to documents with a PMID increased the queries’ sensitivities and decreased their PPVs. We expect this bias to be nondifferential, insofar as it should affect the 3 types of queries in the same way.

The publications with PMIDs 26420387 [38] and 26420598 [83] were written by the same authors and were based on the same search query. This was also the case for PMIDs 26830055 [45] and 26830221 [82]. However, we considered these publications to be independent SRs, insofar as the corresponding GSs were different.

Our results and the literature data provide quantitative information on the use and value of MeSH terms in the queries used for SRs. MeSH terms still appear to be important for achieving a comprehensive SR. Our results also emphasized how difficult it is to build a query for an SR and highlighted the significant variability in the results obtained; the search strategies are a matter of concern for researchers [91-93]. With a view to gaining insights into the possible benefits of MeSH terms for use by less experienced researchers, it would be interesting to conduct a similar study of literature searches performed by clinicians. Finally, our study also highlights that any bibliographic research involves a tedious process of sifting through articles, akin to finding a needle in a haystack. While the authors of SRs perform this task efficiently, inexperienced clinicians might find it discouraging to search for scientific articles. New tools based on network analysis [94] could help these clinicians find relevant articles more quickly.

The objective of this study was to estimate the utility of MeSH terms, selected by authors, in SR queries by analyzing the queries from 59 SRs published in 4 high-impact medical journals in general medicine. Our results revealed that removing MeSH terms from a query decreases sensitivity while slightly increasing the PPV. Queries containing both MeSH and free-text terms yielded more relevant articles but required screening many additional papers. Despite this additional workload, MeSH terms remain indispensable for SRs and can be particularly beneficial for inexperienced clinicians or nonnative English speakers, aiding in constructing more comprehensive queries. However, mixed queries combining MeSH and free-text terms show poor PPV, suggesting the exclusive use of either MeSH terms or free-text terms for rapid reviews.