Real-World Effectiveness of AI-Enabled Clinical Decision Support in Primary Care: Systematic Review

Introduction

AI-enabled clinical decision support systems (CDSS) can provide predictions or recommendations during primary care consultations. Their clinical value depends on whether they improve decisions and outcomes in routine care. Diagnostic accuracy, changes in management, clinician workload, and patient-important outcomes therefore require separate evaluation.

Susanto et al [1] reviewed machine learning CDSS, predominantly in secondary and tertiary care, and found mixed effects on decision-making, care delivery, and patient outcomes. Gomez-Cabello et al [2] described heterogeneous AI-CDSS implementations in primary care. These reviews provide a basis for examining primary care effectiveness while distinguishing systems with learned components from deterministic decision rules.

This review assesses real-world evaluations of clinician-facing AI-enabled CDSS in primary care. We distinguish diagnostic and case-finding performance, changes in clinical decisions or care processes, clinician efficiency, and patient-important outcomes. The support-impact-outcome framework organizes these outcome levels without assuming that an improvement at one level causes improvement at another.

Methods

Overview

We followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 statement [3]. The completed checklist is provided in Multimedia Appendix 1 [4-13]. The protocol was registered with PROSPERO (CRD420261302104). Registration timing and subsequent changes to the methods are described in Multimedia Appendix 1 [4-13].

Operational Definition of AI

AI-enabled systems were defined as those in which a clinically relevant prediction, classification, retrieval, or recommendation depended on a data-derived model, including machine learning, deep learning, natural language processing, or case-based reasoning that generated patient-specific output from prior cases. Multicomponent CDSS were eligible when a learned component formed part of the deployed intervention; effects were attributed to the whole intervention when that component could not be isolated. Deterministic alerts, guideline rules, electronic reminders, and fixed clinical prediction equations were classified separately. Studies of these tools were retained as contextual evidence and were neither formally appraised nor used to support conclusions about AI effectiveness.

Inclusion Criteria

Studies were eligible when they (1) evaluated a clinician-facing CDSS used during routine primary or ambulatory care; (2) included a learned (data-derived) or case-based AI component under the operational definition above; (3) involved licensed clinicians delivering first-contact or continuing primary care, including general practitioners (GPs), family physicians, primary care pediatricians, and allied health professionals such as primary care physiotherapists; (4) evaluated real patient care rather than simulated cases; and (5) reported an evaluable diagnostic accuracy, clinical, behavioral, care process, efficiency, or patient-important outcome against a comparator or reference standard. Deterministic studies meeting the other criteria were retained only for the contextual stream.

Exclusion Criteria

We excluded studies that did not evaluate clinician-facing use in real primary or ambulatory care, including simulations or vignettes, retrospective reader or benchmark studies, model development or validation studies without prospective clinical use, patient-facing tools without clinician-facing decision support, and specialist or hospital-only settings. We also excluded protocols; conference abstracts without a full evaluation; and feasibility-, usability-, or attitude-only reports without an evaluable clinical or care-process outcome. Uncontrolled reports were excluded when no comparator or reference standard allowed an effect estimate.

Information Sources and Search

Five principal bibliographic databases were searched: PubMed, Scopus, Web of Science Core Collection, Cochrane CENTRAL, and CINAHL Ultimate. A supplementary search was run across 10 institutionally available EBSCOhost databases; EBSCOhost is treated as a platform rather than as a database. ClinicalTrials.gov and the World Health Organization (WHO) International Clinical Trials Registry Platform (ICTRP) were also searched for registered and unpublished evaluations. The registered PROSPERO protocol specifies a publication window of January 1, 2011, through December 31, 2025. The upper bound defines a fixed, complete calendar-year publication window; the later search dates allowed retrieval of records indexed after that boundary. No language limit was applied at the search stage; the registered protocol specifies no language restrictions on eligibility.

The search used 3 concepts: AI, clinical decision support, and primary or ambulatory care, with database-specific controlled vocabulary and free text. Trial registries were searched more broadly because registry records contain less standardized descriptive information. Exact executed strings, quotation behavior, filters, search dates, and yields are reproduced in Multimedia Appendix 1 [4-13].

Study Selection and Data Extraction

Titles and abstracts and, subsequently, full-text reports were assessed independently by 2 reviewers. Disagreements were resolved by discussion and, where necessary, adjudication by the senior author. Records lacking an abstract were screened using the title, subject headings, indexing terms, and available record metadata rather than requiring all 3 search concepts to appear in the visible citation. Data extraction captured setting, condition, model class, comparator or reference standard, outcome type, effect estimate, and reported uptake or fidelity. One reviewer extracted the data and a second reviewer independently checked each field against the primary report. Model class assignment and outcome domain coding were performed independently by 2 reviewers using the definitions above, with senior adjudication for borderline systems.

Risk of Bias and Certainty of Evidence

Risk of bias was assessed with the standard Cochrane risk of bias 2 (RoB 2) tool for the individually randomized trial [14], the RoB 2 cluster-randomized extension for the 5 cluster trials [15], ROBINS-I (Risk of Bias in Non-Randomized Studies of Interventions) for nonrandomized intervention studies [16], and QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies 2) for diagnostic accuracy studies [17]. QUADAS-2 risk of bias and applicability were reported separately. GRADE (Grading of Recommendations Assessment, Development and Evaluation) [18] was applied to diagnostic case finding, diagnostic accuracy and screening, clinician efficiency, and patient-important clinical outcomes. Each rating addresses the clinical inference stated in the summary of findings within the evaluated settings; it does not represent a pooled effect across diseases. Differences in populations, interventions, outcomes, and study methods informed the certainty judgments. Care-process studies were synthesized descriptively because their end points did not support a common effect estimate. No meta-analysis was performed because conditions, interventions, and outcomes differed within each body. Selective reporting was considered in the study-level assessments. Publication bias and small-study effects were not formally assessed because each rated body contained at most 2 studies.

Ethical Considerations

This review used published aggregate data only and involved no identifiable participant information or intervention. Under the policy of the Department of General Practice, Beijing Tsinghua Changgung Hospital, secondary analysis of published literature, institutional review board review was not required, and no application was submitted.

Results

Study Selection

The final searches identified 3586 records from the 5 principal bibliographic databases and the supplementary EBSCOhost platform search: PubMed (n=821), Scopus (n=927), Web of Science (n=634), Cochrane CENTRAL (n=111), CINAHL Ultimate (n=890), and EBSCOhost (n=203). CINAHL Ultimate was searched separately and is not part of the 203-record supplementary EBSCOhost total. Of the EBSCOhost records, 183 were exportable, and 20 were screened manually in the platform. Across the exportable database and platform records, deduplication left 2265 unique records; adding the 20 manually screened EBSCOhost records gave 2285 database and platform records screened. The 2 registries returned 537 records (ClinicalTrials.gov n=433 and WHO ICTRP n=104); 469 records remained after duplicate or overlap removal and were screened separately, with no eligible published evaluation identified. A total of 66 reports were assessed at full text, 42 were excluded with reasons, and 24 studies met the review criteria. In total, 10 used AI-enabled systems and formed the appraised evidence population; 14 used deterministic rule-based or fixed-equation logic and were retained only as contextual comparison. Figure 1 shows the study flow, Table 1 classifies the included studies, and Table 2 summarizes full-text exclusions.

‎

Characteristics of Included Studies

The AI-enabled evidence base comprised 10 studies spanning cardiovascular case-finding, dementia detection, melanoma and glaucoma screening, diabetic retinopathy screening, urinary tract infection management, HIV prevention, medication-related fall prevention, childhood asthma, and musculoskeletal pain. The systems included convolutional neural networks, deep computer vision models, natural language processing systems, electronic health record (EHR) risk models, interpretable machine learning classifiers, a case-based reasoning system, and one multicomponent CDSS containing a data-derived prediction model alongside guideline rules. Designs included 5 cluster-randomized trials, 1 individually randomized trial, 2 nonrandomized intervention evaluations, and 2 prospective diagnostic accuracy studies (Table 1).

Risk of Bias

Risk-of-bias judgments are shown in Figure 2 and Multimedia Appendix 1 [4-13]. All 6 randomized trials were judged as having some concerns overall; for the 5 cluster trials, the additional identification or recruitment domain did not change the overall category. Herter et al [9] and Pinto et al [7] were at serious risk under ROBINS-I, driven principally by confounding and temporal change in their nonrandomized comparisons. Under QUADAS-2, Papachristou et al [6] was at high overall risk because of clinician-selected sampling and differential reference verification, with high applicability concern for its restricted lesion spectrum. Jan et al [8] was at high overall risk because only 277 of 414 participants contributed analyzable images; applicability was of low concern for the intended opportunistic primary care screening setting. Judgments from the 3 appraisal tools were kept separate.

‎

Synthesis by Outcome

Case-finding results differed between the 2 randomized trials. AI electrocardiography increased new diagnoses of low ejection fraction (2.1% vs 1.6%; odds ratio [OR] 1.32, 95% CI 1.01-1.61; P=.007), with a stronger effect among patients flagged as high risk (OR 1.43, 95% CI 1.08-1.91) [4]. In contrast, an EHR machine learning passive digital marker for dementia did not increase new diagnosis when used alone (10.3% vs 12.4%; adjusted OR 0.84, 95% CI 0.63-1.11) and did not increase diagnostic assessment (27.8% vs 29%; adjusted OR 0.94, 95% CI 0.72-1.22). The combined arm, in which the marker was paired with a patient-reported instrument, increased new diagnosis to 15.4% (adjusted OR 1.31, 95% CI 1.05-1.64) [5].

Two studies assessed prospective diagnostic accuracy. A deep learning dermoscopy application used prospectively in Swedish primary care reached an area under the receiver operating characteristic curve (AUROC) of 0.960 for melanoma, but all lesions proceeded through standard work-up and the evaluation did not isolate a management effect [6]. In Australian general practice, the glaucoma system achieved an AUROC of 0.80, sensitivity of 65%, and specificity of 94.6% among 277 participants with analyzable images; the report went into the GP consultation, but image acquisition failed for a substantial proportion of recruited participants [8]. Neither study measured patient-important benefit attributable to the AI output.

At the clinician decision and care-process level, results were mixed and often difficult to attribute to the learned component alone. In a controlled before-after study, the interpretable urinary tract infection system was associated with increased treatment success from 75% to 80%, with a larger change in confirmed users (75% to 83%; P<.001) [9]. In the pre-exposure prophylaxis (PrEP) cluster trial, EHR machine learning prompts did not significantly increase initiation of PrEP care overall (6% vs 4.5%; hazard ratio 1.32, 95% CI 0.84-2.10), although a prespecified interaction favored clinicians already caring for people with HIV [10]. In the diabetic retinopathy program, NaIA-RD was deployed before GPs reviewed images and influenced the screening criteria of 3 of 4 GPs; agreement with GPs was at least 94.6% for nonreferral proposals but more variable for referral proposals. Because the study compared long periods before and after implementation without a concurrent control, the change could not be attributed to AI alone [7]. In the SNOWDROP cluster RCT, the multicomponent intervention improved shared decision-making scores for both GPs and patients (P<.001), reduced decisional conflict (P<.001), and improved communication satisfaction, but the primary medication-change analysis was inconclusive (OR 2.09, 95% CI 0.42-10.40) [11].

In a survey nested within the asthma trial, 28 of 42 clinicians responded and reported median estimated record review times of 3.5 minutes with the CDSS and 11.3 minutes without it (P<.001) [12]. This was a comparison of clinician estimates, including estimated time needed without the report, rather than objectively timed work or a randomized comparison of review time. Childhood asthma exacerbations did not differ between supported and usual care (12% vs 15%; OR 0.82, 95% CI 0.37-1.96) [12]. In SupportPrim, global perceived effect at 12 weeks was also not improved (55.4% vs 54.8%; adjusted OR 1.18, 95% CI 0.50-2.78). A clinically important improvement on the patient-specific functional scale occurred less often in the intervention arm (59.7% vs 70.3%; adjusted OR 0.41, 95% CI 0.20-0.85), favoring usual care, although the between-group difference was smaller than the trial’s prespecified 15% threshold for clinical importance [13]. Neither trial demonstrated patient-important benefit from AI-CDSS.

The 14 contextual deterministic studies covered cardiovascular risk and stroke or anticoagulation decisions [19,22,25,32], diabetes and hypertension [20,26], test ordering and kidney failure risk [21,23], liver disease [24,30], cervical screening follow-up [27], opioid use disorder care [28], ambulatory differential diagnosis [29], and general primary care diagnostic and treatment support [31].

Implementation and Potential Harms

Implementation was incompletely measured across the AI-enabled evidence. Herter et al [9] reported a stronger effect among confirmed users; the SupportPrim trial and process evaluation by Granviken et al found low fidelity and use of the tool mainly to reinforce existing practice [13,33]; and Pinto et al [7] and Westerbeek et al [11] evaluated systems embedded in wider clinical workflows rather than isolated algorithmic components. In the study by Jan et al [8], only 66.9% of recruited participants yielded analyzable retinal images. Harms such as overdiagnosis, false positives, automation bias, inappropriate reassurance, and alert burden were rarely prespecified or quantified. Limited reporting prevented conclusions about safety. Per-study implementation observations are reported in Multimedia Appendix 1 [4-13].

Certainty of Evidence

Certainty was low for diagnostic case-finding, diagnostic accuracy and screening, and patient-important clinical outcomes, and very low for clinician efficiency (Table 3 and Figure 3) [4-6,8,12,13]. The care-process category was not assigned a common GRADE rating because it combined treatment success, PrEP initiation, screening and referral decisions, and shared decision-making or medication change.

‎

Discussion

Principal Findings

The 10 AI-enabled studies reported mixed effects across outcome levels. Case-finding improved in the electrocardiogram trial but not with the dementia algorithm alone. Diagnostic discrimination varied by application, and care-process findings depended on the intervention and end point. Neither trial assessing patient-important outcomes demonstrated benefit. These results did not establish a consistent progression from diagnostic performance to improved patient outcomes.

Incomplete uptake, low fidelity, and multicomponent interventions limited interpretation of the observed effects. The studies did not distinguish these factors from model calibration, population differences, task selection, or intervention design. The evidence therefore could not identify a common cause for the differences between diagnostic, process, and patient outcomes.

Interpretation and Implications

Figure 4 separates model output, clinician-facing support, changes in care, and patient-important outcomes. The framework identifies measurements needed to connect these stages. The included studies generally assessed different interventions and populations at different stages, so comparisons across outcome levels could not establish a causal pathway.

‎

Low adherence to recommendations also requires interpretation. Clinicians may modify or reject advice because of patient preferences, comorbidity, competing risks, or information unavailable to the model. Evaluations should record these reasons and assess the appropriateness of decisions, alongside uptake and patient outcomes.

Comparison With Prior Reviews

The mixed effects observed here are consistent with the broader findings of Susanto et al [1] and the heterogeneity of primary care implementations described by Gomez-Cabello et al [2]. This review separates learned and deterministic systems and evaluates certainty according to the outcome measured. It also identifies limited direct evidence connecting changes in care to patient-important benefit.

Limitations

The small evidence base spans different conditions, interventions, designs, outcome definitions, and follow-up periods. These differences limit comparisons and certainty judgments across studies. Embase was not searched. The contextual deterministic studies were not formally appraised. Publication bias could not be reliably assessed in the small outcome bodies. The efficiency finding relied on an unblinded clinician survey with incomplete response.

Conclusions

Evidence published through December 2025 did not establish consistent clinical benefit from AI-enabled CDSS in primary care. Some evaluations reported improvements in case-finding or care processes, but effects varied and patient-important benefit was not demonstrated. Future trials should measure changes in clinical decisions and patient outcomes within the same evaluation, with adequate follow-up and explicit assessment of uptake and harms.