Anticoagulants are cornerstone therapies for stroke prevention in atrial fibrillation and for the treatment and prophylaxis of venous thromboembolism (VTE). However, their complex dosing criteria and monitoring requirements across oral and injectable anticoagulants make them susceptible to medication errors. Warfarin therapy is complicated by numerous drug and food interactions, necessitating regular international normalized ratio (INR) monitoring. Although direct oral anticoagulants (DOACs) have established themselves as the mainstay of anticoagulant therapy owing to their superior safety and convenience over warfarin, precise dose selection based on age, body weight, renal function, and other clinical factors remains essential [1]. Accordingly, prescribing errors remain frequent; a recent systematic review reported that approximately 20% of DOAC prescriptions contain at least one error [2]. Because inappropriate anticoagulant use can lead to serious harm, anticoagulants are classified as high-alert medications in international safety frameworks and in Korea [3-6]. Pharmacovigilance data further indicate that a substantial proportion of bleeding events associated with oral anticoagulants was preventable, mainly due to drug-drug interactions or inappropriate dosing [7].

Medication errors involving anticoagulants are not limited to overdose-related bleeding; underdosing with subsequent thromboembolism is also a critical concern. Off-label DOAC use in patients with atrial fibrillation has been reported in 24% of cases overall, with higher prevalence in Asia (32%) compared with North America (14%) or Europe (22%) [8]. This pattern may reflect heightened concern about bleeding risk in Asian practice settings [9]. However, meta-analyses have shown that off-label underdosing is associated with increased risks of ischemic stroke or systemic embolism without a corresponding reduction in major bleeding, whereas off-label overdosing is associated with higher risks of both major bleeding and thromboembolic events [10,11]. Together, these findings indicate that both underdosing and overdosing are clinically harmful and emphasize the need for accurate, guideline-concordant anticoagulant dosing supported by system-level interventions.

Injectable anticoagulants also present important safety challenges. In particular, low-molecular-weight heparin (LMWH) is prone to prescribing errors because dosing depends on body weight, renal function, and indication. Previous studies have reported medication errors in up to 34% of LMWH prescriptions, with most errors attributable to inappropriate dose adjustments based on body weight or renal function [12]. Given LMWH’s narrow therapeutic window, such errors may lead to treatment failure from underdosing or major bleeding from overdosing, reinforcing the need for structured safeguards to support safe prescribing and monitoring.

Clinical decision support systems (CDSSs) have therefore emerged as key tools to reduce prescribing errors. By integrating clinical information such as diagnoses, laboratory results, and concomitant medications, CDSSs can assess prescription appropriateness and support individualized decision-making, and they have been shown to reduce medication errors and adverse drug events in diverse health care settings [13-21]. Broadly, CDSSs for medication safety can be categorized into rule-based and AI-based approaches, each offering distinct advantages and limitations.

Rule-based CDSSs can reduce prescribing errors by generating targeted, actionable alerts. However, they rely on predefined rules that require continuous curation and maintenance and often fail to capture complex, multifactorial patient contexts that contribute to prescribing errors [22]. Moreover, rule-based systems may generate a high volume of alerts with limited clinical value, contributing to alert fatigue. In South Korea, alerts from the national Drug Utilization Review (DUR) system in tertiary care hospitals have shown an alert override rate of 73%, largely attributed to alert fatigue [23]. Similar studies in other countries have reported override rates ranging from 62% to 95% [24]. Excessive, low-value alerts increase clinician workload and erode trust in CDSSs, underscoring the need to improve alert specificity, prioritization, and clinical relevance.

Machine learning–based CDSSs can provide flexible, data-driven predictions by leveraging complex patterns in large-scale clinical data. Nonetheless, their limited interpretability can impede clinical acceptance, workflow integration, and governance. To bridge the gap between transparency and predictive performance, hybrid CDSSs that combine rule-based logic with machine learning models have recently been proposed. These systems combine the complementary strengths of rule-based and data-driven approaches, enabling reliable detection of well-defined prescribing errors while extending predictive capability to complex clinical contexts not easily captured by fixed rules. For example, a French study demonstrated that a hybrid approach improved both accuracy and reliability in detecting prescribing errors [19].

Despite their potential, few hybrid CDSSs have been developed and evaluated in real-world clinical practice, particularly for high-alert medications such as anticoagulants. To mitigate the risks associated with the complex dosing and monitoring requirements of these agents, we developed a hybrid CDSS to detect and prevent prescribing errors involving anticoagulant therapy. The system integrates rule-based logic with a machine learning model and was implemented in 3 tertiary care hospitals for development, internal validation, and external validation to assess performance and clinical utility. We aimed to evaluate whether the hybrid CDSSs could comprehensively identify anticoagulant prescriptions requiring pharmacist intervention.

We conducted a retrospective study using electronic health record (EHR) data from hospital A, a tertiary academic medical center in South Korea with 1616 beds. Hospital A served as the development site for model development, pilot testing, and internal validation. External validation was conducted at 2 tertiary academic medical centers in South Korea, including hospital B (1245 beds) and hospital C (800 beds). The hybrid CDSS was designed to support pharmacist-led prescription review by identifying orders that may require pharmacist intervention. Reporting of this evaluation study followed the STARE-HI (Statement on Reporting of Evaluation Studies in Health Informatics) guideline, and a completed checklist is provided in Table S1 of Multimedia Appendix 1.

The study was approved by the Institutional Review Board (IRB) of Seoul National University Hospital (IRB number 2307-105-1449). Informed consent was waived due to the use of deidentified retrospective data and the minimal risk posed to patients. Each participating institution obtained separate ethical approval for the external validation (IRB number B-2404-894-403 and IRB number 2024-01-031).

Pharmacists assessed alert appropriateness across 3 domains: technical accuracy, clinical relevance, and clinical usefulness. Before the evaluation, reviewing pharmacists from all sites participated in a training session to harmonize adjudication criteria across institutions. During the session, representative alert cases and the rationale for each adjudication decision were shared and discussed. All alerts were independently reviewed by 2 pharmacists, and any disagreements were resolved through discussion to reach a consensus. Technical accuracy reflected whether an alert was triggered as intended by the underlying rule logic or model specification when applied to the patient’s actual clinical data at the time of prescribing. An alert was classified as technically correct if the triggering criteria were correctly satisfied and the alert rationale matched the corresponding clinical parameters documented in the record. Clinical relevance captured whether an alert represented a potentially meaningful safety concern in the context of the hospital’s routine prescribing practices and local clinical protocols. Alerts were classified as clinically relevant when pharmacist review was warranted to confirm appropriateness, address a plausible medication safety issue, or rule out an exception. Alerts were classified as irrelevant when, despite meeting technical criteria, they did not require further evaluation due to clear contextual justification or lack of clinical significance in the local setting. Clinical usefulness evaluated the downstream value of the alert in supporting pharmacists’ decision-making and workflow. Alerts were classified as useful if they prompted an in-depth review of the medical record or prescription details and contributed to an actionable decision, including contacting the prescriber, recommending changes, or documenting the rationale for continuing therapy after a structured review. Alerts were classified as not useful if they were acknowledged without meaningful assessment, did not change the level of review beyond usual practice, or did not inform any clinical decision.

Additionally, all anticoagulant prescriptions were independently reviewed by at least 2 pharmacists to establish reference labels for model performance evaluation. Model discrimination was assessed using the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve (AUPRC), with additional metrics, including accuracy, sensitivity, specificity, positive predictive value, and negative predictive value. CIs were estimated using the DeLong method for AUROC and exact Clopper-Pearson intervals for proportion-based metrics. The CIs for the AUPRC were obtained by bootstrap resampling with 1000 replications.

The 2 rounds of pilot testing were conducted from November 23, 2023, to November 29, 2023 (excluding weekend days), and on December 8, 2023, respectively. These rounds were conducted to identify technical issues and refine rules prior to deployment. Prescriptions reviewed during the pilot phase were subsequently combined with the complete 2020 development dataset to update the machine learning component of the CDSS, resulting in the final model for deployment. These pilot-tested prescriptions were not included in any of the reported validation datasets. After the application of the finalized CDSS model, internal validation was conducted at hospital A through pharmacist review of all anticoagulant prescriptions issued on 2 predefined dates in 2024: March 26 and April 29.

External validation was conducted after the CDSS had been finalized. Local preprocessing at each external institution was limited to mapping EHR data to the finalized CDSS input structure and did not involve adjudication labels, model retraining, threshold selection, site-specific calibration, or performance optimization. Pharmacist adjudication was conducted independently according to prespecified criteria. External validation was conducted at 2 tertiary hospitals: hospital B and hospital C. Retrospective validation was performed at both sites during routine clinical practice from September 23, 2024, to September 27, 2024, and an additional prospective validation was conducted at hospital C on December 23, 2024.

For model development, we extracted 75,200 anticoagulant prescriptions from hospital A in 2020, of which 812 (1.1%) were classified as prescribing errors. During 2 pilot testing periods, an additional 435 prescriptions were reviewed, and 9 prescribing errors were identified; these data were incorporated into the training dataset for model updating.

For internal validation, 466 anticoagulant prescriptions issued on March 26 and April 29, 2024, were reviewed, with 12 (2.6%) prescriptions adjudicated as prescribing errors. Among 284 DOAC prescriptions in the internal validation set, the automated indication mapping agreed with the documented indication for 252 (88.7%) prescriptions (Table S2 in Multimedia Appendix 1). External validation was conducted at 2 tertiary hospitals. At hospital B, 966 prescriptions were reviewed over 5 days in September 2024, and 82 (8.5%) prescribing errors were identified. At hospital C, 417 and 31 prescriptions were reviewed during the first and second validation periods, yielding 22 (5.3%) and 4 (12.9%) prescribing errors, respectively. The proportions of missing values for each hospital are presented in Table S3 in Multimedia Appendix 1.

The hybrid CDSS integrated a rule-based system with a machine learning model. The initial rule set comprised 38 patient-specific rules addressing diagnosis, body weight, age, renal function, and hepatic function, as well as 1129 rules for contraindicated drug-drug interactions involving anticoagulants.

Across 2 pilot-testing rounds, 667 alerts were generated, of which 583 (87.4%) were judged technically correct. Pilot testing identified several technical issues, including the retrieval of outdated laboratory results and incorrect treatment duration calculations. As a result, 9 existing rules were revised and 6 new rules were added. Prescriptions reviewed during the pilot phase were subsequently incorporated into the training dataset to update the machine learning model. The workflow of the hybrid CDSS is presented in Figure 1, with the system architecture and dashboard screenshots provided in Figures S1 and S2 of Multimedia Appendix 1, respectively.

During internal validation at hospital A, the hybrid CDSS generated 88 alerts, corresponding to an alert rate of 18.9% (88/466). All alerts were technically correct. Of these, 78 (88.6%) alerts were judged clinically relevant, 76 (86.4%) were considered clinically useful, and 12 (13.6%) led to pharmacist interventions. Representative examples of alerts are shown in Table S4 of Multimedia Appendix 1.

In external validation across 3 site-periods, alert rates ranged from 22.6% (7/31) to 32.1% (310/966), with the number of alerts per site ranging from 7 to 310. Across sites, 93.2% (109/117) to 100% (7/7) were technically correct, 78.4% (243/310) to 91.5% (107/117) were clinically relevant, 26.5% (82/310) to 85.7% (6/7) were clinically useful, and 18.8% (22/117) to 57.1% (4/7) required pharmacist intervention (Table 1).

Among rule-based alerts, drug-drug interactions were the most frequently identified category, accounting for 28.6% (2/7) to 47.9% (56/117) of alerts, followed by contraindications and overdosing. The distribution of alert categories varied across institutions (Table 2).

No false negatives were observed for prescriptions requiring pharmacist intervention in the internal or external validation datasets. The AUROC of the hybrid CDSS was 0.916 (95% CI 0.899‐0.933) in internal validation, ranged from 0.871 to 0.880 in the 2 retrospective external validations, and was 0.963 in the small prospective feasibility validation. The AUPRC ranged from 0.568 to 0.833 across validation sites, exceeding the corresponding baseline prevalence at every site (Table 3).

In this multicenter study, we developed and externally validated a hybrid CDSS that integrates a rule-based model and a machine learning model to prevent prescription errors involving anticoagulants. Using pharmacist-adjudicated interventions as the reference standard, the system demonstrated high discriminative performance, with AUROC values of 0.916 in internal validation and 0.871 to 0.963 across external validation sites. Moreover, alerts generated by the system were consistently judged to be technically correct, clinically relevant, and clinically useful. Together, these findings suggest that a hybrid CDSS can enhance prescription review for high-risk medications while maintaining a clinically acceptable alert profile across hospitals with heterogeneous practice environments.

Medication errors involving anticoagulants are common and clinically important. Among various types of errors, inappropriate dosing remains the most frequent and critical concern [2]. Systematic reviews have reported that 13% to 37% of DOAC prescriptions in real-world practice are inappropriately dosed, often because patient-specific factors such as age, body weight, and renal function are not adequately considered [2]. Furthermore, Ko et al [28] identified impaired renal function, advanced age, and prior DOAC exposure as key risk factors for inappropriate dosing. These findings underscore that patients on long-term anticoagulation may be particularly vulnerable to errors, as their clinical parameters, such as renal function and weight, evolve over time. In our study, the proportion of prescriptions classified as prescribing errors varied across institutions and study periods, from 1.1% (812/75,200) in the development set, based on routine pharmacist interventions, to 2.6% (12/466) to 12.9% (4/31) in validation samples. This heterogeneity likely reflects not only true differences in prescribing patterns and patient risk but also differences in review intensity, local workflows, and how intervention-worthy problems are operationalized by pharmacists. Importantly, despite this variability, the hybrid CDSS consistently identified all prescriptions requiring pharmacist intervention, suggesting that it can function as a reliable safety net across diverse clinical settings.

Although DOACs are often perceived as safer and simpler than warfarin because of their fixed dosing, their dose adjustment criteria remain complex and difficult to fully implement within conventional DUR systems. Consequently, off-label or suboptimal dosing is frequently observed in clinical practice. To address these challenges, our hybrid CDSS encoded eligibility, contraindication, and dose-adjustment rules derived from product labeling and major drug information databases, while also incorporating laboratory data and concomitant medication information. By combining explicit rule-based logic with a machine learning model trained on historical pharmacist interventions, the system was able to capture broader aspects of patient context, including patterns that are difficult to formalize as static rules alone.

Across internal and external validation, the hybrid CDSS generated alerts for 18.9% (88/466) to 32.1% (310/966) of anticoagulant prescriptions. Notably, no false-negative cases were observed in the validation sample, suggesting high sensitivity for identifying prescriptions requiring pharmacist intervention within the evaluated periods. Prior CDSS studies illustrate the trade-off between alert burden and safety coverage. For example, MedGuard reported alerts for only 2.36% of prescriptions [13], but did not assess the risk of missed high-risk cases. Conversely, systems such as Check of Medication Appropriateness (CMA) and PharmaCheck produced high alert volumes, with a limited proportion of alerts judged clinically relevant [15,21]. In this context, the hybrid CDSS showed a potentially useful balance between alert burden and safety coverage. This balance was achieved with a moderate alert rate, and no false-negative cases were observed in the validation samples. However, the absence of false negatives should be interpreted cautiously. This finding may partly reflect the short validation periods and the limited number of prescriptions requiring intervention. Further evaluation in larger and longer-term cohorts is needed to determine whether this apparent safety coverage can be maintained in routine anticoagulant prescribing.

In internal and external validations, 26.5% (82/310) to 86.4% (76/88) of alerts were judged clinically useful, and 13.6% (12/88) to 57.1% (4/7) led to pharmacist intervention. These findings indicate that the system not only identified prescriptions warranting review but also captured a substantial proportion of cases in which active intervention was required to ensure patient safety. Although few studies have reported comparable metrics specifically for anticoagulants, prior pharmacist-oriented CDSS evaluations provide relevant context. In PharmaCheck, 20.1% of alerts resulted in pharmacist intervention, whereas only 1.6% did so in the CMA [15,21]. Additionally, the AUROC values observed in our study compare favorably with those reported for a previously reported hybrid CDSS (AUROC 0.81) [19]. While direct comparisons are limited by differences in target medications and outcomes, the relatively high proportion of alerts leading to intervention in our study suggests that the developed hybrid CDSS represents a competitive and clinically impactful approach.

In this study, we implemented a hybrid CDSS that integrates rule-based logic with machine learning models. The system provides explicit rule-based alerts for prescriptions contraindicated by renal function or involving clinically significant drug-drug interactions, while also generating risk-based alerts for marginal dosing patterns and other clinically ambiguous situations not readily captured by predefined rules. This design supports both clearly defined safety checks and broader identification of clinically relevant prescribing risks.

Another important feature of this system is its pharmacist-centered design. Rather than delivering alerts directly to physicians, the CDSS was designed to support pharmacists as primary users who triage, interpret, and contextualize alerts before escalation. Prior work has shown that pharmacist-mediated filtering can substantially increase the positive predictive value of alerts, from 20.1% to 71% in one study of PharmaCheck [15]. In contrast to physician-facing CDSS that often contribute to alert fatigue, our system presents alerts to pharmacists via a dashboard, enabling efficient prioritization of high-risk prescriptions and minimizing unnecessary interruptions to prescribers. In this context, the relatively low positive predictive values of 0.136 to 0.571 can be understood as a consequence of prioritizing comprehensive detection, while the overall alert rate of 18.9% (88/466) to 32.1% (310/966) remains comparable to those reported for other CDSSs. Importantly, this review burden is handled within the established pharmacist triage process rather than being added directly to the prescriber workflow. This workflow design aligns decision support with existing medication review processes and enhances both efficiency and acceptability.

Several limitations should be considered. First, both internal and external validation periods were relatively short, ranging from 1 to 5 days. The reference standard required exhaustive prescription-by-prescription chart review by 2 pharmacists for each validation day. Therefore, validation was deliberately limited to predefined index days to ensure complete manual adjudication. These short validation periods may not fully capture long-term operational challenges such as changes in prescription patterns, system drift, or evolving user behavior. Longer-term prospective evaluations are needed to assess durability and support continuous refinement. In addition, the absence of false negatives in the validation samples should be interpreted cautiously, given the short validation periods, small number of intervention-requiring prescriptions, wide CIs, and the permissive hybrid structure in which alerts were generated when either component flagged an order.

Second, the development dataset was split at the order level rather than the patient level. Therefore, prescriptions from the same patient could have appeared in more than one development subset, which may have led to optimistic development-stage performance estimates. However, the reported temporal and external validation datasets were independent of the development split. Third, the higher proportion of machine learning–based alerts observed at one external site was likely attributable to greater missingness in laboratory data. Alerts triggered by missing information may contribute to alert fatigue if not clearly distinguished from alerts based on abnormal values; however, missingness may also carry clinically meaningful information. For example, missing hematologic or laboratory values may indicate insufficient monitoring in patients receiving anticoagulants, and such alerts may therefore provide a monitoring-related safety message. These findings highlight the importance of explicit alert categorization and complementary strategies to improve data completeness. In addition, alerts related to drug-drug interactions accounted for 29% to 48% of all alerts, yet only a limited proportion resulted in pharmacist intervention. This discrepancy suggests that the frequency of interaction-related alerts may exceed their clinical relevance and contribute to alert fatigue. Stratifying interaction alerts into contraindicated and precautionary categories may improve clinical utility and reduce unnecessary interruptions. Fourth, the system was developed specifically for anticoagulants, limiting its immediate applicability to other drug classes. Nevertheless, successful implementation in this high-risk domain suggests that the underlying framework may be extendable to other high-alert medications with appropriate adaptation. Fifth, reliance on structured data precluded the use of unstructured clinical information such as free-text notes or surgical reports. Although surrogate indicators were used, future work incorporating natural language processing may enhance contextual awareness. Pharmacogenomic determinants of warfarin dosing, such as CYP2C9 and VKORC1 polymorphisms, were also not incorporated, reflecting routine clinical practice in which warfarin is generally initiated at a standard dose and adjusted according to INR monitoring rather than guided by pre-emptive genotyping. Incorporating pharmacogenomic information may further refine the knowledge base in the future. Additionally, we performed both internal and external validation in tertiary care hospitals, and further studies are needed to assess the applicability of the model in other health care settings. Finally, the reference standard was based on pharmacist-documented interventions, which may underestimate the true error rate by missing unrecognized or undocumented errors and introduce label noise. The machine learning component was trained using pharmacist-documented interventions from a single tertiary center with established pharmacist review practices, so the resulting labels may partly reflect a site-specific intervention standard. Because pharmacist intervention practices vary across institutions, the model’s alerts and performance may generalize less well, and site-specific recalibration may be needed before implementation in new settings. More sophisticated integration strategies, such as risk-tiered alerting or dynamic weighting of rule-based and model-based signals, may further improve performance and warrant future investigation. Also, the model was not optimized for precision-oriented metrics, which may leave an unnecessary pharmacist review burden. Future refinement could improve precision while maintaining adequate safety coverage. This study did not evaluate clinical outcomes, including bleeding, thromboembolism, adverse drug events, length of stay, mortality, or actual reductions in prescribing errors. Accordingly, the findings should be interpreted as evidence of the system’s ability to identify prescriptions requiring pharmacist review, and future prospective studies should determine whether this translates into measurable improvements in patient safety.

In conclusion, we developed a hybrid CDSS integrating rule-based logic with machine learning to detect prescribing errors involving anticoagulants and, for the first time in Korea, evaluated it through multicenter external validation. The system demonstrated robust performance, high clinical relevance and usefulness, and a substantial proportion of alerts leading to pharmacist intervention, without missing prescriptions requiring intervention. By centering pharmacists in the alert workflow, the system minimizes unnecessary alerts to physicians while supporting efficient prioritization of high-risk prescriptions. These findings suggest that the hybrid CDSS is a practical and scalable tool for supporting pharmacist-led review and pharmacist workflow efficiency in the management of high-risk medications.