A prospective, multicenter study was carried out at the EDs of University Hospitals Leuven (UZ Leuven) and General Hospital Sint-Jan Brugge-Oostende AV (AZ Sint-Jan Brugge-Oostende), Belgium, from November 2017 to May 2022.

UZ Leuven is a tertiary 1995-bed teaching hospital. The ED consists of an admission and a treatment area (12 boxes), an ambulatory zone, a pediatric zone, and 3 observation care units (30 beds of which 6 are equipped as intensive care beds). On average, 197 patients per day visit the ED, of which approximately 30% (60/197) are hospitalized.

AZ Sint-Jan Brugge-Oostende is a 1182-bed general hospital. The ED consists of an admission and treatment area (11 boxes), an ambulatory zone, a pediatric zone, and 3 beds reserved for critically ill patients. Approximately 92 patients per day visit the ED, of which 38% (35/92) are hospitalized.

Depending on the presumed medical diagnosis, patients are treated by a physician specialized in internal medicine, surgery, emergency medicine, pediatrics, or psychiatry. As part of the obligatory patient assessment, a medication history is obtained by the physician in all patients within 24 hours of the ED visit and is entered in a dedicated medication history module in the EHR. Subsequently, the entered medication history should be electronically validated once considered complete.

Pharmacy, biomedical sciences, or hospital pharmacy students trained in medication reconciliation enrolled planned admission patients on weekdays between 8:30 AM and 5 PM, starting with the patient present the longest and proceeding in descending order. Medication reconciliation was performed based on student availability during their internship. Medication reconciliation was performed independently from the information obtained by physicians. While medication reconciliation was performed between 8:30 AM and 5 PM on weekdays, the sample included patients arriving to the ED during the evening, night, and weekend as a result of the need for observation in the ED or waiting for inpatient beds. Exclusion criteria were (1) patients aged younger than 18 years; (2) patients receiving end-of-life care; (3) patients transferred from another hospital or ward; (4) patients not speaking Dutch, French, or English; (5) patients discharged home or deceased before the medication history was obtained; (6) patients with intentional intoxication; or (7) patients in isolation due to a possible COVID-19 infection.

Every student performing medication reconciliation received formal training by a clinical pharmacist (SDW or GVDS) prior to patient recruitment. Interrater reliability for medication reconciliation was evaluated using a similar methodology as previously described by Pippins et al [1]. Every student had to obtain 4 fictional medication histories through role-play. Results were compared with the best possible medication history as composed by the research team. A cutoff of 90% agreement was set for each fictive case.

Variables were considered for the MED-REC predictor if (1) a significant association between the variable and the occurrence of medication discrepancies had been shown in the literature [21] or if (2) the variable was considered of interest as discussed by our multidisciplinary research team. Furthermore, EHR availability of the variable upon ED visit was required.

Multivariable logistic regression with backward stepwise selection according to the Akaike information criterion was used to select the final model. The presence of at least 1 clinically relevant discrepancy was the dependent variable. Sex, age, language (Dutch, French, or English), residence before ED admission (home, nursing home, or others), transport to the ED (patient’s own transport, ambulance, or emergency physician transport vehicle), ED triage acuity scale using the Emergency Severity Index (ESI), number of drugs documented by the ED physician in the medication history module, number of drugs documented by the ED physician in the medication history module of a certain first WHO (World Health Organization) Anatomical Therapeutic Chemical (ATC) level (ATC A, ATC B, etc), number of high-risk drugs documented by the ED physician in the medication history module, time of ED visit (morning, afternoon, evening, or night), specialty of the physician obtaining the medication history (emergency medicine, internal medicine, psychiatry, or surgery), availability of a regionally shared electronic medication record, and electronic validation of the medication history were investigated as predictor variables.

Continuous predictors were evaluated for nonlinearity using scatter plots, and various transformations were applied to address any nonlinear relationships. However, these transformations did not improve model performance compared with simpler models. Therefore, we decided to forego transforming continuous predictors to maintain interpretability and simplicity.

Missing data were present for the ED triage acuity scale (160/824, 19.4%) and were handled using dummy variable adjustment.

The predictive performance of the model was assessed by measuring calibration, discrimination, and classification in all datasets. Calibration reflects the agreement between model-based predictions and observed outcomes, and was assessed using calibration plots with a target slope and intercept of 1 and 0, respectively. Discrimination refers to the ability of the prediction model to differentiate between those experiencing at least 1 clinically relevant discrepancy and those who do not. Discrimination was evaluated using the concordance index (c-index), for example, the area under the receiver operating characteristic curve (AUROC) for models with binary end points. Bootstrap 95% CIs were calculated. We assessed the following classification measures in various scenarios relevant to daily clinical practice: specificity, sensitivity, positive predictive value, negative predictive value, positive likelihood ratio, negative likelihood ratio, and alert rate. Scenarios were defined by specific probability thresholds, which were used to classify patients predicted to show at least 1 clinically relevant medication discrepancy or not. We investigated the Youden index (maximized sensitivity and specificity), with a sensitivity of at least 80% for hospitals with sufficient staffing resources and a specificity of at least 80% for hospitals with limited staffing resources.

Commonly, the development and validation datasets differ in proportion to outcome events, yielding poor calibration of the original model when applied to new data. Accordingly, adjusting the intercept of the original model to the validation sample can improve calibration [20]. In such a model update, the correction factor for the intercept is estimated in the validation dataset and should be added to the intercept of the original model when applying the model to new patients [22].

The utility of the model for decision-making was evaluated using decision curve analysis [23-25]. In brief, decision curve analysis calculates a “net benefit” for a prediction model by comparing it to (at least) 2 default strategies: exposing either all patients or no patients to an intervention—in this study, medication reconciliation. The net benefit weighs the true positives against the false positives and is calculated using the following equation [23]:

The MED-REC predictor was compared with two existing patient selection algorithms, as used in practice by ED pharmacists and physicians: (1) random selection and (2) patients aged ≥75 years and taking ≥5 drugs.

Data analysis was performed using R (version 4.1.1; R Core Team). The R packages givitiR, pROC [26], and dcurves were used to assess calibration, discrimination, and net benefit, respectively. Descriptive statistics were presented as frequency with percentage for categorical data and as median with IQR for continuous data. Baseline characteristics of the 3 cohorts were compared by chi-square test for categorical data and by the Kruskal-Wallis test for continuous data.

No sample size calculation was performed for model development as we anticipated a large number of inclusions and a high number of discrepancies [19]. A sample size calculation was performed for the temporal validation to estimate the absolute agreement rate with a prespecified precision, defined as the maximum width of the associated 95% CI. In this calculation, we assumed an absolute agreement rate of 70% between the MED-RED predictor and the gold standard, to be estimated with a 95% CI that should have a maximal half-width of 5%. The required sample size was estimated at 333 patients, which also exceeded the minimum of 100 events and nonevents as suggested by the TRIPOD guidelines, assuming a prevalence of 35% as observed in the development dataset [20]. For the geographic validation, we intended to include a pragmatic sample of 100 patients, based on resources available at the external hospital.

The study was approved by the Ethics Committee Research UZ/KU Leuven and AZ Sint-Jan Brugge Oostende (S60638). Written informed consent was obtained prior to inclusion from each patient or relative when the patient was deemed incapacitated. All study data were deidentified. Participation was voluntary and not financially compensated.

In total, 824, 350, and 119 patients were included in the development, temporal validation, and geographic validation dataset, respectively. A flowchart of the study participants is presented in Figure 1. Patient characteristics, drug-related factors, and factors related to ED visits and the medication reconciliation process that were considered candidate variables for the MED-REC predictor are presented in Table 2. Data on the number of drugs before and after medication reconciliation and data on observed medication discrepancies are shown in Table 3. All students achieved the cutoff level of interrater reliability with a mean percent agreement of 96%.

Out of the 824 included patients, 287 (35%) had at least 1 clinically relevant discrepancy, with a total of 475 clinically relevant discrepancies.

After feature selection based on 26 initial variables, the final prediction model contained 8 variables (Table 4). The retained predictor variables were found in similar proportions across all datasets (Table 2). Figure 2A shows that our MED-REC predictor was well calibrated over a broad range of probabilities. The calibration slope was 0.72 and the intercept was 0.03. Discrimination was moderate with an AUROC of 0.66 (95% CI 0.62-0.70; Figure 2D). Classification measures are shown in Table 5. The Youden index, a sensitivity of at least 80%, and a specificity of at least 80% were found at probability thresholds of 0.31, 0.25, and 0.45, respectively.

Out of 350 included patients, 131 (37%) had at least 1 clinically relevant discrepancy, with a total of 205 clinically relevant discrepancies. Excellent calibration was found over the whole range of probabilities with a slope of 1.09 and an intercept of 0.18 (Figure 2B). Discrimination was moderate with an AUROC of 0.67 (95% CI 0.61-0.73), which was not different compared with the development dataset (Figure 2E). Classification measures are shown in Table 5.

Out of 119 included patients, 58 (49%) had at least 1 clinically relevant discrepancy, with a total of 116 clinically relevant discrepancies. The calibration curve is shown in Figure 2C. A slope of 1.35 and an intercept of 0.83 were found. The predictions underestimated the observed outcome probability of having at least 1 clinically relevant medication discrepancy for estimated probabilities between 0.3 and 0.52. Discrimination was moderate with an AUROC of 0.68 (95% CI 0.58-0.78), which was not different compared with the development dataset (Figure 2F). Classification measures are shown in Table 5.

The incidence in the geographic validation dataset was 49% and the mean predicted risk was 35%. Adjustment of the intercept was performed and calibration and discrimination were reassessed. The correction factor was 0.58. The calibration plot of the updated model in the geographic validation dataset showed excellent calibration over the whole range of probabilities (Figure 3). Discrimination of the updated model did not differ from that of the original model (AUROC 0.68).

The MED-REC predictor showed net benefit over the other strategies (treat all, treat none, and patients that are ≥75 years and taking ≥5 drugs) over a range of clinically reasonable threshold probabilities (20%-50%) in the development cohort (Figure 4). Decision curve analysis confirmed the net benefit of the MED-REC predictor over a similar range of threshold probabilities in the temporal validation cohort (Figure S1 in Multimedia Appendix 3). In the geographic validation cohort, the net benefit for the updated MED-REC predictor was demonstrated at higher threshold probabilities, starting from 30% (Figure S2 in Multimedia Appendix 3). The miscalibrated MED-REC predictor did not show a net benefit (Figure S3 in Multimedia Appendix 3).

We developed and externally validated the MED-REC predictor, according to the current standards of predictive modeling [20]. The final model contains 8 variables that can be easily extracted from the EHR upon ED presentation. The MED-REC predictor identifies patients at risk for at least 1 clinically relevant discrepancy with moderate performance and clearly outperforms existing selection strategies. The clinical usefulness of the model was demonstrated through decision curve analysis, which showed net benefit for the MED-REC predictor compared with default strategies and typical selection criteria. While there is no question that conducting a comprehensive medication reconciliation for each hospital admission is of the utmost importance, the stark reality is that this objective is frequently untenable in clinical practice. Hence, our MED-REC predictor can be used to guide the rational use of limited resources at the ED and is especially useful in countries where resources are insufficient.

As an outcome variable, we opted for clinically relevant discrepancies, allowing us to focus on patients with the highest need for medication reconciliation. The current prediction model contrasts with our previously developed model, which screened for any medication discrepancy [19]. After applying the former model in clinical practice, we experienced its lack of clinical usefulness as the model identified many patients with nonclinically relevant discrepancies, for example, discrepancies concerning vitamin supplements.

We validated the MED-REC predictor in both a temporal and geographic validation dataset. Discrimination was moderate and was retained in both external validation datasets, demonstrating the robustness of our MED-REC predictor. The lack of excellent discrimination might be due to other unknown or unmeasured variables not included in our model. However, upon ED presentation, readily available patient information is limited. Calibration was excellent in the temporal validation dataset, whereas underestimation of the observed outcome by the model predictions was observed in the geographic validation dataset, due to a higher prevalence of the outcome in the geographic validation dataset. A simple model update improved calibration in this setting [22,27].

Net benefit was demonstrated over a range of clinically reasonable threshold probabilities (20%-50%) in both the development and temporal validation datasets. Notably, in the geographic validation dataset, the net benefit was observed starting from a higher probability threshold of 30%, which can be attributed to the higher prevalence of the outcome in this cohort (Figure S2 in Multimedia Appendix 3).

Implementation of the MED-REC predictor, a relatively simple prediction model based on readily available variables, is feasible in clinical practice. A precondition is the structural availability of the variables in the EHR in order for the MED-REC predictor to work optimally. The probability threshold can be easily customized to accommodate the available staffing resources. A lower probability threshold will lead to a higher alert rate and fewer false negative results, which might be interesting for hospitals with a large staff available to perform medication reconciliation. A higher probability threshold will be better suited for hospitals with limited staff, hence decreasing the false positive results. The decision curve analysis illustrates how the probability threshold impacts clinical decision-making (Figure 4). The threshold reflects the trade-off between the benefits of true positives and the potential harms of false positives. For instance, a probability threshold of 20% suggests a willingness to perform medication reconciliation on 5 patients to identify the outcome, at least 1 clinically relevant discrepancy, in one patient. This threshold implies that the benefit of detecting one true positive is deemed 4 times greater than the potential (financial) harm of performing unnecessary medication reconciliations [25].

Literature on medication reconciliation risk stratification tools is scarce. Moreover, most published models are not validated at all or are limited to discrimination measurement. Pippins et al [1] published the potential adverse drug event tool based on an observational study on general medical wards. An exploratory model was developed to predict the number of potential adverse drug events, but this model was not validated. Besides, automatization of their tool is limited because not all included predictors are structurally available in the EHR, for example, patient understanding of preadmission medications, and family members or caregivers as the source of preadmission medication information. Ebbens et al [16] developed and validated a risk prediction model to identify patients at risk for medication discrepancies at planned hospital admission. Retained variables were the number of drugs and the presence of cardiovascular and respiratory comorbidities. Good discrimination was found in the development cohort (AUROC 0.75) but was not retained in the external validation cohort (AUROC 0.54). Unfortunately, calibration was not assessed. Recently, Chu et al [15] reported a case series on the implementation of medication reconciliation prediction models in clinical practice. These models were initially developed to identify medication errors upon discharge as part of the MARQUIS2 [28] study and were adapted by removing variables that are not available upon ED presentation. However, the models were not developed based on predictive modeling standards nor was their performance assessed to account for this new setting. Thus, we cannot draw any conclusions regarding the robustness or generalizability of these models. Similar to our study, Damlien et al [17] developed and tested a prediction model to identify patients at risk for clinically relevant medication discrepancies upon ED admission. The authors included a limited sample of 276 patients over the course of 7 weeks, which was used for both development and validation of their model. Four variables were included, for example, sex, age ≥60 years, hospital admission in the last 12 months, and admission reason (malfunction, surgical, or cancer). Using a fixed cutoff value, the model classified patients as having a high versus low risk for discrepancies. An AUROC of 0.7 was found for the development cohort but was not reported for the validation cohort. Again, calibration was not assessed.

Our study has several strengths. First, we included large datasets and applied the current state-of-the-art of predictive modeling. Next to the model development, we performed extensive external validation, including both temporal and geographic validation. As such, we have developed a robust prediction model to identify patients who will benefit from comprehensive medication reconciliation upon ED presentation. Second, we showed how a simple model update can be performed to increase the performance of the model in a distinct setting. Third, we only included candidate predictors that are widely and consistently available within the EHR and accessible upon ED visit. As a result, the MED-REC predictor can easily be implemented in clinical practice. Finally, we chose the presence of at least 1 clinically relevant discrepancy as the outcome variable, thereby increasing the clinical relevance of our model, and allowing us to focus on patients with the highest need for medication reconciliation.

Our study also has some limitations. First, clinical relevance of discrepancies was defined by an in-house expert panel and is therefore open to discussion. Currently, there is a notable lack of consensus regarding the classification of clinically relevant discrepancies. Leading international organizations such as the WHO and accreditation bodies like the Joint Commission International have yet to establish standardized guidelines. Therefore, we emphasize the need for international consensus or standardized indicators to guide the development of future risk prediction models. While our work represents a first step in defining clinically relevant medication discrepancies, external validation by additional experts or institutions (eg, the World Academic Council of Emergency Medicine) is warranted. Second, the requirement of informed consent introduced the possibility of volunteer bias, where individuals who choose to participate may differ systematically from those who do not, possibly affecting the generalizability of our results. However, only a minority of screened patients refused informed consent (ie, 3.4%, 1.3%, and 5% in the development, temporal, and geographic validation cohort, respectively). Third, our enrollment protocol, for example, starting with the patients who have been present the longest, may have introduced selection bias, as they might present with more complex pathology, which might reduce the generalizability of our results. We chose this approach to ensure the inclusion of patients presenting during nighttime or on weekends, aiming to reduce potential bias related to the timing of ED presentation. Although we recognize that this approach may introduce other forms of bias, we think it remains limited. For instance, very complex patients can be both transferred very quickly (eg, a patient having a major cardiovascular event that is transferred immediately to the catheterization laboratory) as well as stay in the ED for an extended duration (eg, a patient with prostatitis who clinically deteriorates and is challenging to transfer). Finally, additional variables, which might further increase the performance of our model, may be missing. Variables that can be assessed in future research include the presence of specific comorbidities and multimorbidity [29], ED crowding, the day of ED presentation and whether it is a holiday or weekend, the number of prior hospital visits and revisits, and the date of the last medication history validation. We did not include these variables in this study because they were either not structurally available in our EHR, not documented, or not prevalent. The presence of multiple comorbidities would likely increase the risk of medication discrepancies due to the complexity of the patient’s medical regimen and the likelihood of polypharmacy. Nonetheless, the number of preadmission drugs and preadmission drugs per ATC code might have partially captured comorbidities in our study. High patient volumes and staff workload during periods of crowding might lead to rushed or incomplete medication reconciliation, increasing the risk of discrepancies.

The probability of having at least 1 clinically relevant medication discrepancy can be calculated by the MED-REC predictor with moderate performance. The MED-REC predictor outperforms medication reconciliation at random or based on other typical selection criteria, offering a guide for the rational use of limited resources at the ED. Depending on available resources, different probability thresholds can be applied to increase either the specificity or the sensitivity of the MED-REC predictor.