Patients who underwent general anesthesia surgery from March 1, 2009, to December 31, 2019, at seven academic hospitals of the Catholic University of Korea (Seoul St. Mary’s, Yeouido St. Mary’s, Uijeongbu St. Mary’s, Eunpyeong St. Mary’s, Bucheon St. Mary’s, St. Vincent, and Incheon St. Mary’s Hospitals) were included. The exclusion criteria were as follows: operation-related criteria were operation duration under 1 hour or duration not available, cardiac surgeries, operations of brain death donors, nephrectomies, and kidney transplant operations; and renal function-related exclusion criteria were patients with a history of renal replacement therapy, preoperative serum creatinine (sCr) ≥4.0 mg/dL or estimated glomerular filtration rate (eGFR) <15 mL/min per 1.73 m², elevation of preoperative sCr more than 0.3 mg/dL or 1.5 times within 2 weeks before surgery, and patients without preoperative or postoperative sCr values (Figure 1).

The study was approved by the institutional review board of the Catholic University of Korea, College of Medicine (XC20WIDI0080) with waiver of consent due to the retrospective study methods. This study was not registered as it is a retrospective observational study. This report has been written according to the recently updated TRIPOD+AI (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis+Artificial Intelligence) statement [14].

Postoperative AKI was defined as AKI that developed within 30 days after surgery, using the Kidney Disease: Improving Global Outcomes (KDIGO) criteria [15]. Stage-1 AKI was defined as sCr 1.5 to 1.9 times above baseline or an increase in sCr ≥0.3 mg/dL; stage-2 AKI was defined as sCr 2.0 to 2.9 times above baseline; and stage-3 AKI was defined as sCr more than 3 times above baseline, ≥4 mg/dL, or the initiation of renal replacement therapy (hemodialysis, peritoneal dialysis, or continuous renal replacement therapy). We did not use the urine output criteria of KDIGO, as previous studies suggested that the threshold of oliguria for postoperative AKI may be different from those of other AKIs [16,17] and due to the lack of data. This definition of postoperative AKI was used to create the supervised learning dataset of those with or without postoperative AKI.

We collected data on demographic characteristics; underlying clinical diseases; preoperative laboratory data; preoperative medication; and surgical characteristics such as expected operation time, the day of operation (weekday or weekend), and the department of surgery. The underlying diseases of subjects were determined using the International Statistical Classification of Diseases and Related Health Problems, 10th Revision (ICD-10) codes of principal and secondary diagnosis. Comorbid diseases and ICD-10 codes are shown in Multimedia Appendix 1. Preoperative medications included RASi (angiotensin converting enzyme inhibitor [ACEi] or angiotensin II type 1 receptor blocker [ARB]) or NSAIDs. Preoperative eGFR was calculated from the Chronic Kidney Disease Epidemiology Collaboration equation [18]. BMI was calculated as the patient’s weight in kilograms divided by height in meters squared (kg/m²).

Anonymized data was extracted from the Catholic Medical Center (CMC) Clinical Data Warehouse, which is separately generated and managed redundantly from the electronic medical record systems of eight affiliated hospitals of the College of Medicine, the Catholic University of Korea [5] and processed using R software (version 3.6.3; R Foundation for Statistics Computing). The 38 variables included in the final analysis are shown in Table 1. In cases where laboratory tests were conducted multiple times before surgery, we selected the most recent preoperative values, taken closest to the time of surgery, to ensure the data accurately reflected the patient’s latest clinical status. Data artifacts and extreme values were set to the 1st percentile and 99th percentile, and missing values were filled using multiple imputation by chained equations (MICE) [19]. MICE was used to provide more accurate estimates of the missing variables with the correlation of missing variables to other existing data points [20]. We excluded variables with more than 40% missing data, following common practice [21-23]. The rates of the missing data for the variables are shown in Multimedia Appendix 2. Nonbinary data were one-hot encoded, a method for rearranging categorical data into binary variables, and numerical data were normalized using min-max scaling. This would convert all numeric values between or equal to a value of 0 and 1. Min-max scaling is given by:

One-hot encoding, min-max scaling, and dataset splitting were accomplished using the Scikit-Learn library (version 0.24.2) [24]. These steps are required to improve the performance of machine learning models and training stability. Because there was a small percentage of AKI events (3.3%), there was an extreme class imbalance in the dataset. Such imbalances can cause a falsely elevated accuracy and adversely affect machine learning training [25]. To help overcome this issue, the AKI training dataset was augmented using an oversampling method by synthetic minority over-sampling technique (SMOTE), which has been shown to improve imbalanced class classifications (using imblearn library version 0.8.0) [26,27].

Various machine learning methods were used to create the model, which was trained and evaluated using Python (version 3.8.5; Python Software Foundation). Machine learning methods commonly used in health care were applied [28,29]. Models applied were logistic regression, decision tree, random forest, naïve Bayes (using Scikit-Learn library version 0.24.2) [24], light gradient boosting machine (GBM; using lightgbm version 3.2.1) [30], and deep neural network (DNN; using Keras library version 2.5.0) [31]. The strengths and weaknesses of each model have been summarized in Table 2.

For the deep learning model, the structure that was chosen was a model that had an input layer of width 50 (to account for the 40 inputs and to include the one-hot encodings); 3 hidden layers with a width of 64, 32, and 32; and a single output node. The configurations of the different models are shown in Multimedia Appendix 3. The training of the models was done on a machine with Intel Xeon Gold 6240R (8 cores) at 2.40 GHz, with 64 GB of RAM, Windows 10 Enterprise Build 17763. To analyze the statistical performance of the models for postoperative AKI prediction, we assessed the area under the curve (AUC) of the receiver operating characteristic (ROC) curve, accuracy, precision, sensitivity (recall), specificity, and F₁-score. To determine the optimal thresholds for ROC-AUC analysis and the calculation of sensitivity and specificity, we used the Youden index (Youden J statistic). This index is defined as:

Alternatively, it can be expressed as the maximum value of true positive rate – false positive rate, which was the criterion applied in our models to identify the threshold value. This method ensures a balanced trade-off between sensitivity and specificity, as described in Schisterman et al [37].

Statistical analysis was performed using SAS software (version 9.4; SAS Institute). Continuous variables were presented as means and SDs for data with normal distribution and presented as medians and IQRs for data with nonparametric distribution. After the distribution of data between the two groups was determined, they were compared using an independent t test (2-tailed) or Wilcoxon rank sum test. Categorical data were presented as percentages, and a comparison between the two groups was performed using the chi-square test or Fisher exact test. To determine the risk factors for AKI, we used the logistic regression model. Multivariable analysis using logistic regression was performed on variables with a P value <.20 on univariable analysis [38]. The results are presented as odds ratio with 95% CI. P values <.05 were considered significant.

A total of 439,072 surgery cases from seven academic hospitals of the Catholic University of Korea were included in the study (Figure 1). After the exclusion of patients according to the exclusion criteria mentioned above, a total of 239,267 cases were included in the final analysis. Among these, 7935 (3.3%) AKI events occurred. Baseline demographics of patients with and without AKI are shown in Table 3. Significant differences were observed in all baseline characteristics between the two groups. Patients with AKI were older, with a higher percentage of those of the female sex, and had a lower BMI, a higher percentage of smokers, and higher baseline systolic and diastolic blood pressure. The AKI group also showed a higher percentage of all preexisting comorbidities (CKD, diabetes, hypertension, coronary artery disease, cerebrovascular disease, chronic obstructive pulmonary disease, or liver cirrhosis); more frequent usage of RASi (ACEi or ARB) and NSAIDs; and a higher percentage of patients undergoing general surgery, neurosurgery, and thoracic surgery. Laboratory results of the AKI group showed lower levels of hemoglobin, serum albumin, and eGFR, and higher baseline sCr and C-reactive protein levels. Variable selection was performed based on these clinical characteristics using logistic regression (clinical parameters are shown in Table 4 and laboratory parameters are shown in Table 5). During feature selection, the hematocrit variable was removed because it had >0.9 correlation with preoperative hemoglobin levels.

The dataset was divided into the training set (80%) and the test set (20%). The training set (n=191,413) and test set (n=47,854) were balanced for outcomes and randomly assigned. To prevent outcome prediction bias, the testing subset was only evaluated after the model had been finalized. Predictive features were blinded during the outcome assessment phase. The loss function graph and AUC graphs of the training and validation sets for the DNN model are shown in Figure 2. Performance of the training and test sets of the DNN model is also presented in Multimedia Appendix 4. The performances of the different models are shown in Table 6.

We hypothesized that a simple system not using too many variables, for example, fewer than 20 variables, would be more practical to use in a clinical setting. Therefore, we evaluated model 2 and model 3 using multiple machine learning methods. Model 2 included 11 variables that were used in the classification system developed by Park et al [9], including age, sex, emergency operation, operation duration, diabetes, ACEi or ARB usage, blood levels of albumin, hemoglobin, sodium, eGFR, and urine dipstick protein. In this model, light GBM (AUC=0.81) and DNN (AUC=0.8) showed the highest performance. Model 3 included variables that were found significant on multivariable analysis, including age, sex, systolic blood pressure, diastolic blood pressure, operation duration, eGFR, blood levels of creatinine, albumin, sodium, potassium, chloride, glucose, lactic dehydrogenase, and urine dipstick protein. In this model as well, light GBM (AUC=0.825), and DNN (AUC=0.811) showed the highest performance. Model 1 included all 38 preoperative variables and surgical characteristics, on which light GBM (AUC=0.836), and DNN (AUC=0.832) demonstrated the best prediction performance once again. The ROC-AUC for model 1 of the different AKI prediction models is shown in Figure 3.

To enhance clinical applicability, a nomogram was created based on a simplified logistic regression model, focusing on 8 key predictors: age, gender, albumin, hemoglobin, sodium, operation duration, eGFR, and urine protein. The nomogram was created by making modifications to a Python nomogram library called simpleNomo [39]. It provides a graphical tool for clinicians to estimate the risk of AKI in individual patients by integrating these factors. This approach allows quick risk stratification, clinical decision-making, and targeted interventions [40]. The nomogram is shown in Figure 4.

Finally, our postoperative AKI prediction tool, the CMC-AKIX, was developed using all 38 variables. Therefore, the DNN model 1 was developed into a user-friendly website, which can be accessed on the web [41] (shown in Figure 5). This was created using Flask and hosted on a Google Cloud Virtual Machine.