This multicenter, retrospective study was conducted across the general wards of 3 different hospitals. Data were collected from patients admitted to Korea University Guro Hospital and Anam Hospital between January 1, 2015, and December 31, 2021 (internal cohort), and from Soonchunhyang University Cheonan Hospital between March 1, 2016, and March 31, 2021 (external cohort). Patients younger than 19 years, those with fewer than 3 SCr measurements during their hospital stay, and those with an estimated glomerular filtration rate (eGFR) ≤60 mL/minute on the first day of SCr measurement were excluded.

AKI was defined based on the Kidney Disease: Improving Global Outcomes (KDIGO) criteria as follows [1].

The KDIGO criteria are widely used; however, they are sometimes modified based on the characteristics of the cohort or the objectives of the study [28]. Compared with ICU patients, general ward patients are in relatively better condition with less frequent measurements of vital signs and laboratory data. Additionally, individuals with healthy kidneys and low SCr levels can be identified as having AKI due to simple fluctuations [14]. There may be more bias in real-time predictions, as SCr estimation is more frequently performed for patients with missing SCr values. Therefore, to prevent these issues, we restricted the application of baseline SCr levels and refined the labeling. To improve the labeling criteria, baseline SCr was defined as the lowest SCr measured within the previous 7 days. If the patient’s baseline SCr was ≥0.3 mg/dL lower than the median and recent SCr values, it was excluded as a baseline to avoid errors due to SCr measurement inaccuracies or temporary fluctuations caused by medication, measurement error, and other factors. AKI was defined as a minimum increase of 0.3 mg/dL to ensure appropriate identification without mislabeling due to minor variations [2,29,30]. AKD was defined as the persistence of AKI for more than 7 days. Cases in which SCr did not return to <1.5 times the baseline SCr within 7 days were identified as AKD [21,29-32]. None of the previous definitions or studies have taken these aspects into consideration.

To develop an early prediction model for AKI occurrence, data from 1 to 3 days before AKI onset were labeled as 1, and the remaining data were labeled as 0. The data from the day of AKI onset were not used. For patients who did not develop AKI, all data were labeled as 0. Days without SCr measurements within 7 days were excluded from training and evaluation. Additionally, patients with ambiguous AKI labeling were excluded from the training and evaluation. Data from the day of AKI onset were used to predict AKD progression. Patients were labeled 0 or 1 based on whether they recovered within 7 days post AKI onset. Patients with insufficient SCr measurements after AKI onset were excluded. Multimedia Appendix 1 illustrates the previous KDIGO criteria, the improved AKI criteria proposed in this study, and examples of AKI and AKD labeling.

The data collected from the electronic health records contained numerous missing values, including measurement values, times, and specific variables. To address this issue, data were summarized at 24-hour intervals, a method validated in previous studies [33]. Vital signs, laboratory data, and variables such as nephrotoxic drugs (eg, nonsteroidal anti-inflammatory drugs, nephrotoxic antibiotics [ANTIs], and cytotoxic chemotherapeutic agents), vascular imaging examinations, surgeries under general anesthesia, contrast-enhanced computed tomography (CECT), and ICU transfers within the past 7 days were collected.

Vital sign data measured multiple times within 24 hours were summarized using mean, maximum, and minimum values, and the number of measurements. The laboratory test results were determined based on recent measurements. Additionally, variables such as nephrotoxic drugs, vascular imaging examinations, surgeries under general anesthesia, CECT, and ICU transfers within the past 7 days were included. eGFR was calculated using the Chronic Kidney Disease Epidemiology Collaboration 2021 equation [34]. The “BUN/Cr ratio” was defined as blood urea nitrogen (BUN) level divided by serum creatinine level.

Robust scaling was applied to all continuous variables and one-hot encoding was used for categorical variables. Approximately 120 features were extracted from the electronic health records, including basic patient information, vital sign data, laboratory test results, and other factors. Through a comprehensive literature review and consultation with specialists, a feature selection process was undertaken [6,23,35], and 42 features were ultimately selected based on their correlation coefficients and missing value ratios.

To handle outliers, the data distribution for each feature was reviewed along with the individual patient records. Outliers for some numerical variables were determined using histograms and quantiles with input from clinical experts. The missing values were imputed in 2 stages. First, where feasible, missing values were replaced with previous values to maintain data continuity. For variables with low missing values (less than 20%), the multiple imputation by chained equations method was used [36,37]. For variables with a missing rate exceeding 20%, the missing indicator method was used to denote missing values as unknown [38]. Numerical variables were categorized into 2 to 4 groups based on data distribution and expert consultation, and missing values were assigned to the “missing” category. For laboratory test results not subjected to missing indicators, changes in each variable were calculated by subtracting the median of the previous values from the current value (Multimedia Appendix 2).

Various traditional machine learning models have been used, including logistic regression [39], random forest [40], eXtreme gradient boosting [41], light gradient boosting machine [42], and categorical boosting (CAT) [43]. For detailed model training procedures (Multimedia Appendix 3). The model evaluation metrics included accuracy, precision, recall, specificity, F₁-score, the area under the receiver operating characteristic (AUROC), and the area under the precision-recall curve. The primary outcomes were the development of models for early AKI prediction within 3 days and its progression to AKD, along with the establishment of a framework. Simulations of the designed framework were repeatedly performed using the external validation cohort by dividing the period into 1-year increments and assessing the generalization performance of the models across different institutions and over time.

Descriptive statistics were used to present the baseline differences between the days with and without AKI. The distribution of continuous and categorical variables was expressed as means and SDs and counts and percentages, respectively. Normally distributed continuous variables were evaluated using 2-tailed t tests. For those that did not follow a normal distribution, the Mann-Whitney U test was used. The chi-square test was used to analyze categorical variables. Statistical significance was set at P<.05. Calibration plots were used to assess the agreement between the predicted probabilities and observed outcomes [44]. The Cox proportional hazards model was used to compare hazard ratios (HRs) between patients with actual AKD and those predicted by the model [45].

The study was conducted in accordance with the ethical principles of the Declaration of Helsinki and was approved by the institutional review boards of Soonchunhyang University Cheonan Hospital, Korea University Anam Hospital, and Guro Hospital (approvals 2019-10-023, 2023AN0145, and 2023GR0425, respectively). The need for individual consent was waived due to the retrospective nature of the study and the use of anonymized clinical data. To ensure privacy and data security, only fully anonymized data were used, and all analyses were conducted within a designated secure environment with restricted access. No financial compensation was provided to participants, as the study was purely observational and used deidentified retrospective data. All processes adhered to the guidelines developed for machine learning model development in the biomedical field [46] and followed the STROE (Strengthening the Reporting of Observational Studies in Epidemiology) guidelines for observational studies.

Figure 1 is a flowchart of cohort formation and labeling. To compare the previous labeling criteria [7,32,47-49] with the labeling criteria used in this study, we developed and evaluated the model using the same methodology. Additionally, we calculated and presented the HRs for a 30% or 40% reduction in eGFR at the time of AKI onset (Figure 2 and Multimedia Appendix 4). In Table S7 in Multimedia Appendix 4, individuals with a baseline SCr<0.6 mg/dL were included, leading to the classification of AKI in patients with very low changes in SCr or very low baseline SCr levels. Additionally, subtracting the baseline median from the SCr at this point yielded a median of 0.10 (IQR 0.00-0.20).

Figure 2A illustrates this trend. Of the 2509 patients, 2242 (89.36%) had a baseline SCr<0.6 mg/dL at the time of AKI occurrence. The remaining 267 patients experienced decreased SCr levels before being diagnosed with AKI. This trend is illustrated in Figure S5 in Multimedia Appendix 4. Additionally, of the 2072 patients whose AKD status could be determined, only 50 (2.41%) progressed to AKD, and only 1 patient experienced a more than 30% decrease in eGFR within 30 days of AKI onset.

The final analysis included 95,555 and 39,513 cases in the internal and external cohorts, respectively. Within these cohorts, AKI was identified in 7658 (8.1%) and 2898 (7.3%) patients, respectively. The median length of hospital stay was 11 (IQR 7-20) days for the internal cohort and 10 (IQR 6-18) days for the external cohort. The median number of days to AKI occurrence from admission was 8 (IQR 3-17) days for the internal cohort and 7 (IQR 3-16) days for the external cohort (Multimedia Appendix 5). Tables S9 and Table S10 in Multimedia Appendix 5 present the basic statistics of patients with and without AKI. Comparing the days on which AKI occurred to the days on which it did not, there were statistically significant differences in most characteristics between the internal and external cohorts. However, differences in blood pressure and sodium and chloride levels were observed between the internal and external cohorts.

The models were evaluated based on the results of 5-fold cross-validation, with all performance metrics presented at a cutoff of 0.5. The CAT model demonstrated a strong predictive capability (AUROC=0.9134). The logistic regression model performed poorly (AUROC=0.7754). Model evaluation was conducted through both internal and external validations, showing a slight decrease in performance based on the AUROC (range 0.0097-0.0216). Table 1 presents the performance evaluation results of the early AKI prediction model. The hyperparameters used in the model are listed in Table S11 in Multimedia Appendix 5.

Figure 3 shows the Shapley additive explanations (SHAP) analysis for the early AKI prediction model. The interpretation of the CAT model using SHAP values indicated that the most important features were the SCr level and its changes. Other significant features included heart rate, BUN level, activated partial thromboplastin time (aPTT), total bilirubin level, and BMI, all of which increased the AKI risk. Conversely, lower eGFR, platelet count, body temperature, and pH were associated with a higher risk of AKI. Figure S7 in Multimedia Appendix 5 shows the results of testing the probability of the early AKI prediction model. Calibration plots were generated for both the internal and external cohorts. The slope of the calibration plot was 1.15 for the internal cohort and 1.08 for the external cohort, indicating a good calibration of the model's predicted probabilities. Additionally, the model probabilities were evaluated by dividing them into five groups, which demonstrated excellent calibration. A box plot of the probabilities in relation to the AKI status showed a clear distinction between AKI and non-AKI, further validating the predictive value of the model (Figure S8 in Multimedia Appendix 5).

For the analysis of AKD progression, 5429 and 1998 patients in the internal and external cohorts, respectively were included after excluding those with unclear AKD status. Among them, 896 (16.5%) patients in the internal cohort and 287 (14.4%) in the external cohort were diagnosed with AKD. The median duration from AKI to discharge was 10 (IQR 4-23) days in the internal cohort and 9 (IQR 3-21) days in the external cohort. For patients who recovered within 7 days post AKI, the median recovery time was 5 (IQR 3-7) days in both cohorts (Figure S9 in Multimedia Appendix 6). The characteristics of progression to AKD in each cohort are presented in Tables S12 and S13 Multimedia Appendix 6. Variables that demonstrated statistical significance in both cohorts included hemoglobin, albumin, SCr, eGFR, glucose, alanine aminotransferase, blood sugar, calcium, urine specific gravity, C-reactive protein (CRP), ANTIs, and surgery. Some variables such as sex, diastolic blood pressure (DBP), white blood cell (WBC) count, BUN/Cr ratio, and total carbon dioxide showed different trends between the cohorts.

The CAT model outperformed the other models in both the internal (AUROC=0.8202) and external validations (AUROC=0.7833). The external validation showed a slight decrease in performance based on the AUROC (range 0.0262-0.0416). Table 2 presents the performance evaluation results of the early prediction model for AKD. The hyperparameters used in the model are listed in Table S14 in Multimedia Appendix 6.

Figure 4 shows the SHAP analysis of the early-stage AKD prediction model. For AKD prediction, an increase in SCr level at the time of AKI was a strong predictor of delayed recovery. Exposure to ANTIs, high DBP, CRP, aPTT, low eGFR, low pH, and elevated alkaline phosphatase were associated with an increased risk of AKD. Patients who underwent surgery or CECT tended to recover quicker from AKI.

Figure S10 in Multimedia Appendix 6 shows the results of testing the probability of the early AKD prediction model. Calibration plots for both the internal and external cohorts were generated to assess model performance. The slopes of the calibration plots were 1.27 for the internal cohort and 1.20 for the external cohort, indicating a good calibration of the model’s predicted probabilities. The evaluation of the model probabilities by box plots showed excellent calibration, similar to that of the early prediction model for AKI (Figure S11 in Multimedia Appendix 6).

The framework to assist in managing AKI is shown in Multimedia Appendix 7. Figure 5A presents the framework simulation results for the early AKI prediction model. The designed framework was simulated using an external validation cohort at yearly intervals of 5 years. Among the 7284 patients included annually in the external validation cohort, 501 (6.9%) experienced AKI. Of these, 439 (87.68%) were predicted at least 1 day in advance. The F₁-score showed a decreasing trend after 2018 compared to evaluations between 2016 and 2018, with a minimum of 0.39 and a maximum of 0.46. Figure 5B presents the framework simulation results for the early prediction model for AKD. Excluding patients with insufficient SCr tracking from an average of 410 patients annually, 64 (15.6%) did not recover from AKI within 7 days. Among them, approximately 38 (58.4%) were predicted early.

In this study, we developed a framework to assist in the management of AKI. The early AKI prediction model demonstrated high performance with an AUROC of 0.9053, while the early AKD prediction model achieved an AUROC of 0.8202. External validation results showed excellent model performance, although slight variations over time were observed, likely due to changes in disease incidence rates. Our study represents an integrated effort to identify patients with AKI early and, in cases where AKI occurs, classify them into high-risk and low-risk groups. Furthermore, we successfully refined and applied the KDIGO criteria to retrospective data, addressing its limitations and enhancing its applicability for AKI-related research in general ward patients.

Furthermore, our model predicted AKD in patients with AKI and showed an adjusted HR of 2.03 (95% CI 1.38-3.00) for events where eGFR decreased by 30% within 30 days from the onset of AKI. This adjusted HR was higher than that of the model developed using previous criteria, which had an HR of 1.66 (95% CI 1.17-2.34). This indicates that the refined criteria more precisely identify high-risk groups and suggests that the model developed using these criteria performs better. The AKI incidence was higher using the previous criteria, whereas the AKD incidence was higher using the refined criteria. This indicated that the previous criteria identified milder cases. Despite including milder cases, the model developed using the refined criteria showed a higher risk of a poorer prognosis than the previous criteria. Among patients tracked for more than 30 days after AKI, there were 934 and 482 patients before and after criteria advancement, respectively, with 138 and 103 showing poor prognosis. In other words, before the criteria advancement, 35 more patients with a poor prognosis were identified, but 452 additional AKI cases were identified. As shown in Figure S5 in Multimedia Appendix 4, many patients who met only the previous criteria included those whose AKI status was difficult to determine owing to the significant sensitivity to fluctuations. Regardless of how the baseline SCr level is imputed, it is important in general wards to set a minimum increase criterion when applying a relative standard. Additionally, it is crucial to ensure that the baseline SCr level does not become too low when the SCr decreases. Advancements in labeling criteria should consider both patient prognosis and clinical settings. Excessive identification of AKI can lead to the detection of more patients with poor prognoses but may also result in inefficient allocation of medical resources.

Using the SHAP, factors contributing to AKI and AKD were identified and quantitatively measured. These results are largely aligned with the trends suggested by previous studies on risk factors and variable tendencies. Factors indicating poor patient status, such as low albumin or high alkaline phosphatase levels, were associated with increased AKI and AKD risks. Specifically, albumin is considered a critical biomarker closely related to renal function and may decrease in conditions associated with liver dysfunction [50]. Elevated WBC, heart rate, body temperature, and respiratory rate may indicate infection in patients [51], while the use of nephrotoxic drugs such as ANTIs, nonsteroidal anti-inflammatory drugs, and cytotoxic chemotherapeutic agents increases the risk of AKI [52,53]. Analysis of SHAP for AKD risk factors indicated that patients with increased WBC and high BT tended to recover renal function relatively quickly. Resolving infections appears to reduce the risk of AKD development in patients with AKI. ANTIs have emerged as significant risk factors for AKI and AKD. The difficulty in discontinuing these drugs due to patient conditions may exacerbate negative outcomes. Patients undergoing surgery are closely associated with AKI, although AKI following general anesthesia appears to show transient SCr fluctuations and overall health improvement following surgical resolution [54]. Renal function emerged as a crucial factor, consistent with existing clinical studies that state that baseline renal function is well-known as a major factor in AKI [55]. Additionally, higher age, heart rate, respiratory rate, total T-bil, aPTT, and CRP showed increased AKI risk, indicating that poorer patient condition may correlate with increased AKI incidence.

Similarly, poor renal function has been identified as a major risk factor for AKD. Elevated DBP reflects a tendency toward increased blood vessel volume after AKI, suggesting a possible correlation between increased volume and AKD [56,57]. AKI accompanied by surgery may recover relatively quickly due to transient hemodynamic changes during surgery, while cardiac surgery is known to have a relationship with AKI [58]. Contrast-induced nephropathy often shows peak levels approximately 3-5 days after exposure and often returns to baseline within 7-14 days, indicating relatively good recovery [59,60]. Conversely, ANTIs are associated with intrinsic AKIs such as acute tubular necrosis and slower recovery despite expected renal function impairment [61-63]. A high urine specific gravity suggests dehydration, which can often be corrected through fluid supplementation alone, leading to fast recovery [64,65]. Indicators reflecting infection showed that higher values tended to be associated with good renal recovery, while the significant infection marker CRP yielded ambiguous results. Because the interpretation of our model aligns with that of previous clinical studies, the patterns learned by the model are reasonable. Although biomarkers such as cystatin-C are commonly suggested in various studies, they were not used in this study due to their low measurement frequency. Future research should expand the features used in this study to include such biomarkers [66-68].

This study had several limitations. First, owing to the lack of consensus on the AKI recovery criteria, the definitions had to rely on previous research findings. Second, there was a lack of data related to dialysis or kidney transplantation, which was addressed by excluding patients with an eGFR<60 mL/minute. As a result, caution is required when applying and interpreting the model for patients with preexisting chronic kidney disease or poor kidney function from the initial stages of hospitalization. However, the primary aim of this study was to predict “unexpected AKI.” Therefore, the analysis focused on patients with relatively preserved kidney function, who were considered to be at lower risk of AKI. Third, the study did not account for interventions or treatments before or post-AKI, which is crucial because patients’ preexisting conditions are significant factors in AKI and AKD. Fourth, in this study, the model was applied only in cases where AKI occurrences could be clearly identified, specifically when baseline SCr could be estimated at a specific time point and SCr measurements were available at that time. However, the proportion of AKI labels may differ from the actual occurrences. Therefore, despite the frequent measurement of SCr, caution is needed when applying and interpreting the model in situations where SCr measurements have not been conducted. Fifth, although our study used data extracted from different hospital information systems in various regions, it predominantly included data from Korean individuals. Therefore, we could not sufficiently consider racial diversity. Since there may be various differences, including kidney function, depending on ethnicity, future studies should include a multiethnic population.

This study introduces a machine learning framework aimed at assisting in the early management of AKI in general ward patients. To develop the model, we used retrospective data from general wards, refining the operational definition of AKI and externally validating our approach. Our findings demonstrate that AI-driven methods can enhance risk stratification and enable timely interventions. Beyond improving predictive accuracy, this study underscores the potential of AI to streamline clinical workflows, optimize resource allocation, and ultimately reduce the burden of AKI-related complications. Integrating such models into routine hospital practice may support proactive decision-making, allowing physicians to implement tailored interventions based on individual patient risk profiles. Future research should focus on prospective validation, real-time clinical integration, and incorporating additional biomarkers to improve model generalizability and clinical relevance.