With funding from the Canadian Institutes of Health Research and the US National Institutes of Health/NLM, we next started analyzing index and abstract terms in MEDLINE to see if we could identify (and then build) “canned” searches that would retrieve only the studies with the strongest methods (eg, randomized controlled trials for studies of treatment). This would allow for content searches to be limited to articles with a higher likelihood of being rigorously performed. We were successful with this endeavor and have continuously conducted research to improve these “clinical queries” using advanced information analysis and retrieval methods, including new work with AI.

To accomplish this, we developed a method to evaluate the performance of Boolean search terms and combinations, termed “hedges,” to retrieve target articles, which represents an early natural language processing application. The method is based on a manual search of 160 clinical journals for the year 2000, with 49,028 articles classified by article format, interest to human health care, and purpose category, which were critically appraised by highly trained staff with expertise in health research methods. The resulting Hedges dataset was used to test thousands of combinations of search terms using a diagnostic accuracy test approach [15]. The Hedges strategies retrieve original and review articles [16] for a range of purposes such as treatment [17], prediction guides [18], etiology [19], prognosis [20], and diagnosis [21]. Database-specific strategies are available on our website [22], with several available through PubMed, MEDLINE, and EMBASE as Clinical Queries. In 2013, we reported on the robustness, assessed over 10 years, of the strategies for retrieving relevant articles in PubMed [23].

McMaster PLUS (Premium LiteratUre Service) curates high-quality evidence and is comprised of a series of steps in a process referred to as the Health Knowledge Refinery (HKR) (Figure 2). The HKR was designed to distill the flow of articles from a broad selection of clinical journals into a refined product of preappraised literature to support clinicians through several evidence services and products.

The current PLUS process integrates several HiRU innovations. First, nightly automated searches of ~125 journals (selected from a critical appraisal of >800 clinical journals) are filtered using highly sensitive Hedges search strategies developed by the unit in the early 2000s to identify systematic reviews, evidence-based guidelines, and original studies addressing questions of treatment, prevention, quality improvement, economics, diagnosis, etiology, prediction guides, and prognosis.

Second, the filtrate is further refined using a recently developed BioBERT (Bidirectional Encoder Representations from Transformers for Biomedical Text Mining)–based machine learning model that classifies articles for meeting explicit criteria for research rigor and clinical relevance. The model maintains 99% sensitivity (recall) and high precision, reducing the work required to manually appraise articles by 60% [24].

Third, research associates manually critically appraise the filtered articles using established standards for scientific rigor. From a list of 63 clinical disciplines, they select the relevant medical disciplines and indicate if the article is also relevant to rehabilitation professions or nursing. Article assessments are then reviewed by a clinician with expertise in research methods.

Fourth, the McMaster Online Rating of Evidence (MORE) system provides postpublication clinical peer review to further refine the HKR output to the articles that are important for consideration in clinical practice [25]. Qualifying articles are automatically sent to registered clinicians for each pertinent discipline identified during the critical appraisal step. MORE raters are a crowd of ~6000 health care providers practicing worldwide, who rate the articles on 7-point scales for relevance to their practice and newsworthiness (defined as useful new information for physicians). Physicians [26], nurses [27], and rehabilitation practitioners [28] are invited to join MORE via specific links.

Fifth, the final refined HKR filtrate of articles that have relevance and newsworthiness scores ≥4 is sent out via email alerts; added to the PLUS database; and made available through searchable interfaces such as ACP JournalWise, Evidence Alerts, ACCESSSS, and other products and services to support a range of clinical knowledge users (Table 1). We provide content to publishers for updating evidence-based textbooks, which we customize under the auspices of McMaster University, a not-for-profit, publicly funded university. Usage of HiRU products remains strong. Across several services, we have >250,000 registered users with >25,000 new registrations in 2023.

Finally, through PLUS, selected articles that are highly rated for clinical relevance and newsworthiness to practicing clinicians and that are of interest to the broad readership of Annals of Internal Medicine are summarized and included in ACP Journal Club. This EBM-focused enterprise has been active since 1990 and has evolved over time. Originally, article selection was done through a manual search of the contents of printed journals by trained research associates with articles that met the methods criteria shared through fax machines. This process evolved to reviewing the online table of contents and sharing through email. In 2014, we developed an in-house web-based infrastructure that allowed for automated retrieval of article titles and abstracts from PubMed, a collection of a range of data elements entered by research associates during critical appraisal, automated email-based alerting for clinical editors and MORE raters, and a collection of ratings. This also allowed us to track articles that were not applicable to the HKR (eg, not related to human health care, basic science, and methodology studies), those that were relevant but did not meet methodologic criteria, and those below clinical relevance and newsworthiness thresholds for alerting. Through this process, we have curated a dataset of almost 200,000 articles that have been reviewed by human experts, along with various associated data elements gathered at the time of publication.

The Hedges dataset provided the HiRU with opportunities to collaborate with external research partners. The dataset has served as a validated reference standard of high-quality studies and has been used to build and test more advanced literature retrieval models; the search strategies have also been used to identify articles to build comparison datasets. Historically, the retrieval models included conventional methods such as Boolean search filters and citation-based algorithms. Over time, machine learning approaches, particularly supervised models trained using these data, have been effective in retrieving high-quality clinical studies from the biomedical literature [30]. Some examples include the study by Aphinyanaphongs et al [31] in 2005, which used articles abstracted in ACP Journal Club (as a PLUS derivative) and machine learning to automatically construct filters identifying high-quality, content-specific articles in internal medicine. Building on this work, using the Hedges dataset, Kilicoglu et al [32] experimented with 3 supervised machine learning methods (naïve Bayes, support vector machine, and boosting), and obtained comparatively better results.

Advancements in machine learning have dramatically improved computer capabilities through deep neural networks. In 2018, Del Fiol et al [33] used the Hedges dataset to develop a model that was perhaps the first to investigate the use of deep learning techniques to identify reports of scientifically sound studies in the biomedical literature. In 2020, Afzal et al [34] used an optimized multilayer feed-forward neural network model, the multilayer perceptron, for identifying scientifically sound studies and filtering out others. Both studies used the PubMed Clinical Queries filter with a “narrow” scope, favoring high precision over high recall [33,34].

Through the HKR, our dataset of classified and appraised articles has grown, and we have expanded our machine learning capabilities. We recently used the data to train machine learning models in-house, achieving both high recall (sensitivity) and precision. We assessed the efficacy of advanced deep learning models that include BERT (Bidirectional Encoder Representations from Transformers) and its variations such as BioBERT, BlueBERT, and PubMedBERT [35]. In 2023, a state-of-the-art model named DL-PLUS, trained using the dataset, excelled in classifying articles for meeting rigor and clinical relevance compared with competitors [24]. This represents our initial phase of machine learning work, with plans to improve the classification of articles by purpose category (eg, treatment, diagnosis) and rigor to provide more targeted results for searchers.

The incorporation of LLMs into health care is rapidly growing, and EBM is no exception. LLMs have demonstrated remarkable success in various downstream tasks, including information extraction and evidence summarization of clinical studies and bodies of literature [36,37]. To enable the uptake of evidence into practice, LLMs have potential to summarize individual studies and bodies of evidence. At the HiRU, we have conducted pilot testing of AI-generated summaries to gauge how well they include EBM-pertinent details such as a clear research objective, details on the methodology, effect sizes, and conclusions for clinical application. Analogous to our early focus on ensuring ready access to interpretable findings and identifying the best-quality research, of key interest is developing and testing prompts (much like search queries) that task the AI tools with returning the necessary information and assessing the factuality of generated summaries. Our near-future plans include contributing to the methods of ensuring the trustworthiness of evidence summaries generated by LLMs [38], with a focus on a clinical audience. Our extended plan includes developing custom LLMs at the HiRU, contributing to other tasks of the evidence ecosystem such as evidence appraisals.

The majority of machine learning models, particularly deep learning models, are inherently “black boxes,” concealing internal details about the decision-making processes [39]. As a result, determining the true effectiveness of an AI model becomes challenging. In some cases, bias may inherently exist in the dataset (eg, it does not accurately represent the overall population), and machine learning engineers may unintentionally introduce bias (eg, during class balancing through sampling). In a recent model training experiment, we observed that when we attempted to balance classes with an oversampling method, the result was a highly accurate yet poorly calibrated model. Subsequent evaluations using calibration methods revealed that the model with the initial unbalanced data was, in fact, well-calibrated. Therefore, responsible AI practice is crucial, particularly in digital health applications, due to the potential ethical issues associated with AI technologies. The increasing reliance on AI in health care has highlighted the importance of addressing concerns such as biases, discrimination, errors, and lack of transparency in outcomes. Implementing responsible AI practices in this context becomes paramount, emphasizing ethical principles and human values to minimize biases, enhance fairness, ensure interpretability, and ultimately prevent adverse consequences on human and societal well-being [40]. Our focus on responsible AI for futuristic AI models will target 2 key areas: prioritizing the examination of datasets to identify biases related to coverage, correctness, and fairness, while concurrently enhancing the interpretability and explainability of AI model prediction workflows, thereby clarifying the decision-making process.