Data for development and internal validation were collected from longitudinal observational EHR data collected between April 2003 and December 2020 from Seoul National University Bundang Hospital (SNUBH) and externally validated with EHR data from Seoul National University Hospital (SNUH) collected between October 2004 and December 2020. SNUBH and SNUH are university-affiliated tertiary hospitals located in the Seoul metropolitan area and the capital, respectively. Both hospitals offer inpatient, outpatient, and emergency department (ED) services. EHR data collected from both hospitals included the patients’ demographics, diagnoses, drug prescriptions (for outpatients), drug administrations (for inpatients and patients in the ED), operations, vital signs, and laboratory test results. The type of data sets captured in the 2 hospitals were similar. In this study, EHR data from each institution were converted to the OMOP CDM and used in the OHDSI’s large-scale data analytics framework. The OMOP CDM provides a homogenous format for health care data and standardization of underlying clinical coding systems, which enables sharing of the analysis code across participating data sets in the OHDSI data network [10].

Since the data sources were de-identified, the study protocol was reviewed and approved with a waiver of informed consent by the institutional review boards (IRBs) of each participating site which included the Seoul National University Bundang Hospital (X-1912-652-904) and Seoul National University Hospital (E-2011-013-1169). Our study strictly adheres to the guideline about machine learning predictive models in biomedical research [11].

Our retrospective study target cohort consisted of all surgical inpatients aged >65 years who underwent general surgery at the Department of Surgery within 14 days of admission. We developed 4 types of prognostic models for predicting postoperative delirium, prolonged postoperative stay, composite outcome of all-cause mortality and ED visits, and prolonged hospital stay from admission to discharge using SNUBH CDM data.

To predict postoperative delirium, prolonged postoperative stay, and the composite outcome, the prediction time (t=0) and time-at-risk start date of the prediction window were set as the surgery date. The time-at-risk end date for postoperative delirium and prolonged postoperative stay prediction was set as the discharge date, and the time-at-risk end date for the composite outcome was defined as 90 days after surgery.

For the prediction of prolonged hospital stay, the prediction time and time-at-risk start date were set as the admission date and the time-at-risk end date was the discharge date.

The predictor observation time window was defined as –365 days to –1 day relative to the time-at-risk start date for all 4 outcomes. We excluded patients for whom the observation period was less than 1 year. Figure 1 shows the study design for predicting different postoperative outcomes.

The primary endpoint was a composite of all-cause mortality and ED visits up to 90 days after the operation date. For the ED visits, we only included patients who were admitted to the ED for over 24 hours. In previous studies, postoperative complications were usually reported within a period of 30 to 90 days after surgery [12,13]; in this study, a 90-day time-at-risk was used in consideration of the number of outcomes required to learn the model. The secondary outcomes were postoperative delirium, prolonged postoperative stay ≥75th percentile [13-15] (ie, ≥8 days in the SNUBH data set), and prolonged hospital stay of ≥21 days. Postoperative delirium was operationally defined as the initiation of antipsychotic medications (eg, haloperidol, quetiapine, or olanzapine) after surgery in patients who had not been prescribed these medications at any point during the entire observation period. Because the length of hospital stay is one of the main indicators of medical efficiency and hospital management, this study also defined prolonged hospital stay based on the data of hospitalization as a widely used performance indicator.

The covariates used in the predictive model included the following: demographics (sex and age), prior conditions (condition record and condition group era), number of distinct conditions, oral medications, procedures and observation (family history and medical history data), measurement value, number of hospital visits for different visit types, and the CHADS₂-VASc (congestive heart failure, hypertension, age ≥75, diabetes, stroke [doubled], vascular disease, age 65-74, and sex category) score observed 365 days prior to the time-at-risk start date.

An additional analysis was performed using surgery information as a covariate. The prediction performance was evaluated when the surgery concept itself or surgical risk groups (intermediate- or high-risk surgery vs low-risk surgery) were added as predictors into the model. The surgical risk groups were categorized using a previously established risk-stratification model [15].

The standardized open-source OHDSI patient-level prediction package was used to develop and validate the model using observational health care data [16,17]. We developed a prognostic model using the SNUBH CDM development data set. The SNUBH CDM data set was split into training (80% of the data) and testing (20% of the data) sets to perform internal validation of the developed model. For 80% of the training data set, we used a 5-fold cross-validation for hyperparameter tuning and eventually obtained an optimal prediction model [18]. Using the OHDSI patient-level prediction framework, the least absolute shrinkage and selection operator (LASSO) logistic regression, gradient boosting machine, random forest, Adaboost, and decision tree models were developed in this study. The LASSO logistic regression model was used to develop predictive models using regularized logistic regression with an L1 (LASSO) prior, which applies equal shrinkage to the coefficients and enables automatic feature selection and better prediction accuracy owing to its reduced variance [19]. In the hyperparameter grid search, the default setting of OHDSI patient-level prediction was used (see Multimedia Appendix 1 for detailed grid search settings).

To evaluate the models, model discrimination was assessed using the area under the receiver operating characteristic curve (AUC), and model calibration was assessed by inspecting a calibration plot. The generalizability of the model was evaluated by performing internal validation using SNUBH test data and external validation using SNUH data for the model trained with SNUBH CDM data.

This study included data from 27,197 and 32,857 patients from the SNUBH and SNUH databases, respectively. The baseline demographic and outcome characteristics of the target population in the 2 databases are presented in Table 1. The SNUBH and SNUH data sets had an average age of 72.94 (SD 5.48) years and 72.17 (SD 5.54) years, respectively, with men accounting for 54.3% (n=14,769) and 54.15% (n=17,792) of patients, respectively. A similar distribution was found for severity and length of stay. Postoperative delirium occurred at an incidence of 2.81% (n=764) and 1.63% (n=536) in the SNUBH and SNUH populations, respectively.

The number of people eligible for inclusion in the target population, the outcome count, and the number of people lost due to each inclusion step for developing the 4 prognostic models for the different outcomes at SNUBH are presented in Multimedia Appendix 2.

The overall discriminative performance of the different prognostic models for the 2 data sets is described in Table 2. The calibration of the different models is presented in Multimedia Appendices 3-6 for each adverse outcome. The primary outcome of composite outcome of all-cause mortality and ED visits occurred in 11.4% (n=2800) and 6.7% (n=2120) of the patients from SNUBH and SNUH, respectively; postoperative delirium occurred in 2.8% (n=764) and 1.6% (n=536) of patients, respectively; prolonged postoperative stay occurred in 24.1% (n=6563) and 23.8% (n=7826) of patients, respectively; and prolonged hospital stay occurred in 6.8% (n=1853) and 6.4% (n=2102) of patients, respectively.

Among the 5 predictive models, LASSO logistic regression generally showed the best overall performance in internal and external validation across all adverse outcomes. The receiver operating characteristic curve of LASSO logistic regression across internal and external validations is presented in Figure 2. For the primary composite outcome, the AUCs (95% CI) of the model were 0.723 (0.701-0.744) and 0.703 (0.692-0.714) for internal validation and external validation, respectively. When we compared the age or Charlson comorbidity index, the LASSO logistic regression model showed better performance in predicting a composite outcome of all-cause mortality and ED visits (Figure 3). For postoperative delirium outcomes, the AUCs of the model were 0.754 (0.713-0.794) for SNUBH data and 0.754 (0.727-0.772) for SNUH data. For prolonged postoperative stay, the AUCs of the model were 0.813 (0.800-0.825) for SNUBH data and 0.747 (0.741-0.753) for SNUH data. Regarding prolonged hospital stay, the AUCs of the model were 0.770 (0.749-0.792) for SNUBH data and 0.707 (0.696-0.718) for SNUH data. In all 4 outcomes, the random forest model showed the highest performance with an AUC of 0.950-0.987 in training; however, it was found that the performance was lowered to 0.677-0.708 in external validation.

When surgery information was added as a predictor, overall AUC performance improved for all outcomes. Table 3 shows the performance of the LASSO logistic regression model when surgery predictors were added. The highest performance was observed when the surgery concept itself was added as a predictor. In particular, the performance was improved when the surgery concepts were included for the prolonged postoperative and hospital stays.

For the primary outcome of the composite outcome, of the 14,361 candidate predictors in the SNUBH data set, 315 (2.2%) predictors were selected through LASSO regression. The covariates selected for the predictive model and their covariate values are presented in Multimedia Appendix 7. The 5 negative predictors with the strongest association with the primary outcome were measurement values of albumin, prothrombin time, PR interval, condition of “inguinal hernia,” and drug of azelastine. The 3 positive predictors with the strongest association with the primary outcome were ED visit count during the observation period, measurement values of segmented neutrophils per 100 leukocytes in the blood, and the measurement value of glucose.

To predict postoperative delirium, the final model included 141 of 14,687 (1.0%) candidate predictors (Multimedia Appendix 8). The 3 strongest positive predictors were age, condition era group of “malignant tumor of esophagus,” and the measurement value of glucose.

For the prediction of prolonged postoperative stay, 465 of 14,687 (3.2%) candidate predictors were chosen (Multimedia Appendix 9). Measurement values of albumin, condition of “inguinal hernia,” and procedure of “CT of thyroid with contrast” were negatively associated with prolonged postoperative stay, while condition of “malignant tumor of esophagus” and procedure of “plain film of head” were positively associated.

In the predictive model that analyzed prolonged hospital stay, 237 of 14,181 (1.7%) candidate predictors were selected (Multimedia Appendix 10). Measurement values of albumin, condition of “hernia of abdominal wall,” and “neoplasm of breast” were negatively associated with prolonged hospital stay, while measurement values of total bilirubin, C reactive protein, and condition of osteitis were positively associated.

This study demonstrated the feasibility of developing patient-level predictive models to identify adverse outcomes after surgery among older adults. The LASSO logistic regression model performed well, with an internal AUC (95% CI) of 0.723 (0.701-0.744), and also applied well to another data set with an external AUC of 0.703 (0.692-0.714) to predict a composite outcome of all-cause mortality and ED visits.

The covariates selected by our model included variables that correlated with the outcomes of older patients undergoing surgery in previous studies. In the patient-level prediction model for the 3 outcomes, a low serum albumin level was selected as the strongly correlated covariate. Low albumin levels were associated with postoperative mortality, ED visits, postoperative delirium, and prolonged hospital or postoperative stay. This is consistent with the results of previous studies. Low serum albumin concentration has a clear relationship with all-cause mortality in older patients and is a strong prognostic parameter for postoperative outcomes [20-23]. Covariates associated with postoperative delirium were also consistent with previous research on advanced age and operation-specific risk [24]. We also confirmed that surgical disease was the strongest risk factor for a prolonged hospital stay. Breast cancer or hernia presented with shorter hospital stays, and patients with pancreatobiliary-related disease (elevated bilirubin) and C reactive protein levels required a longer time to discharge.

This study bears several strengths. We developed and validated our patient-level predictive model using 2 large data sets covering a wide variety of covariates, including demographics, medical conditions, and prior health behaviors, that were collected through routine care. As comprehensive evaluation among older patients undergoing surgery is necessary, additional effort is required for a comprehensive geriatric assessment. The prediction model using indicators collected during routine care was easier to implement.

Our developed model, presented in Table 2, did not include surgery type or surgical risk as a covariate. Although we observed some improvement when incorporating surgery concepts or surgical risk, as shown in Table 3, the difference in the composite outcome of all-cause mortality and ED visits was not significant. Surgeons can estimate the level of risk associated with a particular surgery based on their experience. Therefore, a risk stratification model that reflects patients’ characteristics is likely to be more useful to surgeons. Despite not including any information about surgery, our risk stratification model that accounts for patients’ characteristics may have significant practical value and can be applied widely in various hospitals regardless of the surgery type or risk.

However, this study also bears several limitations. First, since only CDM-based outcomes were measured, the mortality and ED visit rates may have been underdetected. Second, the incidence of delirium (1.6%-2.8%) may have been underestimated compared to previous results from a meta-analysis, as it was counted based on medication administration for symptom control [25]. Among patients older than 60-70 years, the incidence of postoperative delirium has been reported to be 10%-20% depending on the patient and surgical risk [26]. Furthermore, if delirium has occurred, the drug may not be used without risk of self-harm. Therefore, our operational definition of delirium could have underestimated its actual occurrence. The differences in delirium incidence between the 2 hospitals could be attributed to the differences in the clinical practice policies for delirium even though the patients’ ages and comorbidity statuses at baseline were similar between both hospitals. Third, there was a difference in the model prediction accuracy during external validation depending on the outcome. In our study, the external predictive value of the model for postoperative delirium, prolonged postoperative stay, and prolonged hospital stay was well-maintained; however, it was poor for a composite of mortality and ED revisits in the external validation. This is probably due to the hospital operation characteristics not being reflected in the model or missed outcomes rather than the patient characteristics since there was a difference in the outcome ratios of the 2 data sets (11.4% and 6.7% for SNUBH and SNUH, respectively). Additionally, we could not analyze out-of-hospital mortality and reported a composite outcome within 90 days. The predicted 30-day or 90-day mortality was not presented as a result in this study because a meaningful model could not be developed; such a model required the outcomes of more than 1000 patients [27], which our data set lacked. The composite outcome within 30 days also failed to show an AUC of 0.7 or higher in the internal and external validation; therefore, future studies with larger data sets are required.

We developed and evaluated various patient-level predictions based on routinely collected EHR data that can identify older patients with higher risks of adverse outcomes after surgery. This was successfully developed from observational and OMOP CDM data and was easily transported to different institutions and externally validated. A next possible step may be to apply the model to EHRs, which can assist in decision-making, additional perioperative evaluation, and supportive care to prevent adverse outcomes after surgery.