This retrospective cohort study involved 3 independent tertiary hospitals—Wuxi People’s Hospital, Nanjing First Hospital, and Sir Run Run Hospital affiliated with Nanjing Medical University. Data were collected from January 2016 to December 2017. The inclusion criteria included patients aged 18 years and older who were diagnosed with CAP during hospitalization. The study exclusion criteria were (1) patients diagnosed with stage 4-5 chronic kidney disease (CKD) or receiving regular renal replacement therapy, (2) patients with a history of kidney transplant, (3) patients with a hospital stay of less than 24 hours or exceeding 90 days, (4) patients who had been treated by other medical institutions within 14 days before hospitalization, (5) patients who relied on mechanical ventilation and palliative care for advanced tumors on a chronic basis, (6) occurrence of AKI within 24 hours of admission or 7 days thereafter, and (7) patients with peak Scr levels less than 53 μmol/L or Scr measurements fewer than 2 times between 1 year before hospitalization and discharge or fewer than 1 time during hospitalization.

The primary outcome for developing the ML models was AKI in patients with CAP. AKI is identified based on the Scr level, as stated by the Kidney Disease Improving Global Outcomes (KDIGO) [8]. It is defined as an increase in Scr to ≥1.5 times baseline within 7 days of admission, or an increase in Scr ≥0.3 mg/dL within 48 hours. The baseline Scr level was determined as the average within 7 to 365 days before hospitalization. If baseline Scr is not recorded, the first Scr value upon admission is used as the baseline Scr. Urine output was not considered for AKI diagnosis due to missing data [9]. Estimated glomerular filtration rate (eGFR) was calculated using the Chronic Kidney Disease Epidemiology Collaboration (CKD-EPI) creatinine equation [10]. Proteinuria is defined as “The presence of at least 1+ protein in urinalysis.”

The dataset was divided into 2 parts: one from Wuxi People’s Hospital for training and internal validation (2568/3211, 80% for model development and 643/3211, 20% for validation) and the other from the multicenter patients for external validation.

Baseline characteristics and clinical data of patients within 24 hours of admission, including 62 indicators, were obtained from the electronic record system. Variables with over 15% (9/62) missing values were excluded, and the remaining variables were imputed using the missForest method. Outliers were identified and treated as missing values. The data of each feature were standardized using the z score method. Variables with strong collinearity (variance inflation factor [VIF] value≥10) were discarded using the VIF. The synthetic minority over-sampling technique (SMOTE) was applied to address data imbalance and improve the prediction efficiency of the models.

Feature importance was evaluated using the sklearn library method. Features were sorted in descending order based on their scores. A sliding windows sequential forward feature selection technique method based on random forest (RF) and out-of-bag (OOB) filtering was used to select features related to AKI. The optimal number of features is determined by finding the subset that results in the lowest OOB error (or the highest OOB score). After feature selection, 5 ML algorithms, including logistic regression (LR), RF, support vector machine, extreme gradient boosting (XGBoost), and deep forest (DF) models, were used to construct the prediction models. A detailed description of ML methods was given in Multimedia Appendix 1.

External datasets were used to validate the models, and their performances were evaluated using various metrics such as the AUC, specificity, sensitivity, positive predictive value (PPV), negative predictive value, F₁-score, expected calibration error (ECE), calibration curve, and decision curve. A 1000-bootstrap method was used to determine CIs by sampling with replacement on the prediction indices.

Shapley additive explanations (SHAP) is a method that constructs an additive interpretation model. It considers all features as “contributors” and calculates the marginal contribution of each feature when added to the model output. SHAP aims to provide explanations for “black box models” at both global and local levels, making the model more understandable and applicable.

On the other hand, local interpretable model-agnostic explanations (LIME) modify a single data sample by adjusting the eigenvalues and observing its effect on the output. LIME acts as an “interpreter” by explaining the prediction of each data sample. In LIME, a set of interpretations represents the contribution of each feature to the prediction of a single sample.

To compare baseline characteristics across different datasets, we used 1-way ANOVA and chi-square tests. Statistical significance was determined using a 2-sided P value of less than .05. The ML models were created using Python (version 3.9.7; Python Software Foundation) with packages such as scikit-learn, NumPy, SHAP, and pandas, as well as R (version 4.1.0; R Core Team) with the glmnet package.

The study was approved by the institutional review boards of Wuxi People’s Hospital (KY22073), Sir Run Run Hospital (2021-SR-042), and Nanjing First Hospital (KY20230410-01-KS-01). The study was conducted in accordance with the declaration of Helsinki. Due to its retrospective design, the ethics committees from Wuxi People’s Hospital, Sir Run Run Hospital, and Nanjing First Hospital waived the need for informed consent. Data can be analyzed secondary without additional consent. The personal information of patients was kept confidential throughout the entire study. Since the experimental design did not involve any intervention for the patients, no compensation was paid to the patients in the study.

The study comprised a total of 6371 patients, with 3211 (50.4%) patients assigned to the training and internal validation groups, and 3160 (49.6%) patients assigned to the external validation group (Figure 1). Out of the total patients, 2566 (40.2%) were female, with a mean age of 67.0 (SD 17.6) years and a baseline Scr level of 72.5 (SD 31.1) μmol/L. Table 1 illustrates the differences between the training and validation sets.

Among the 6371 patients, 1006 (15.8%) developed AKI, with the median time for AKI diagnosis being day 3 (IQR 2-4) post admission. A comparison between patients with and without AKI is presented in Table 2. Patients with AKI exhibited higher age, temperature, heart rate, breathing rate, and lower diastolic blood pressure levels. They also had a higher incidence of hypertension, CKD, cardiovascular disease, cerebrovascular diseases, acute respiratory failure, and acute cardiac dysfunction and a higher rate of mechanical ventilation compared with those without AKI. Furthermore, patients with CAP having AKI demonstrated significantly elevated baseline Scr, white blood cell count, neutrophil count, C-reactive protein (CRP), and blood urea nitrogen (BUN) levels compared with those without AKI (all P<.05). Conversely, baseline eGFR, platelet count, hemoglobin, and blood albumin levels were substantially lower in the non-AKI group (all P<.05).

In the training model, 9 variables with over 15% (9/62) missing values were excluded from the final cohort. These variables included blood lactic acid, ferritin, glycosylated hemoglobin C, blood oxygen partial pressure, blood carbon dioxide partial pressure, sputum pathogens, complement C3, complement C4, and procalcitonin. Among the remaining 53 variables, 3 had VIF values≥10. These variables were inhaled oxygen concentration, oxygen inhalation status, and vasoactive drug application. After eliminating variables with collinearity, the remaining 50 features (including age, sex, smoking status, and severe pneumonia), comorbidities (including history of chronic lung disease, CKD, malignancy, hypertension, diabetes, cardiovascular disease, chronic liver disease, and cerebrovascular disease), complications (such as septic shock, acute respiratory failure, and acute cardiac dysfunction), vital signs (including temperature, heart rate, respiratory rate, systolic blood pressure, and diastolic blood pressure), treatment on the day of admission (including noninvasive ventilation, mechanical ventilation, antiplatelet therapy, anticoagulation, renin-angiotensin-aldosterone system [RAAS] inhibitors, proton pump inhibitors, statins, diuretics, nonsteroidal anti-inflammatory drugs, insulin, calcium channel blockers, beta-blockers, and corticosteroids), and laboratory tests (including baseline Scr, baseline eGFR, CRP, urine protein, serum albumin, white blood cell count, platelet count, hemoglobin, neutrophil count, monocyte count, lymphocyte count, BUN, alanine aminotransferase [ALT], aspartate aminotransferase [AST], serum potassium, serum chloride, and serum sodium) were evaluated using the sliding windows sequential forward feature selection technique method.

The model generated an OOB value for each included feature. Figure 2 indicates that as the number of features increased to 11, the OOB value (0.579773) decreased substantially, and subsequently, the OOB value remained stable with increasing features. When the number of features reached 40, the OOB value (0.0340017) reached its lowest point. Therefore, to facilitate clinical application, we selected the first 11 features as the final model variables. These variables included sex, temperature, breathing rate, diastolic blood pressure, CRP, albumin, white blood cell count, platelet count, hemoglobin, neutrophil count, and BUN. Figure 3 illustrates that the BUN and neutrophil count consistently ranked among the top 5 indicators in all 5 models, suggesting their significant contribution to the occurrence of AKI.

A total of 643 patients were included in the internal validation, with an AKI incidence of 13.5% (n=87). Figure 4A illustrates the AUC of the 5 models in the internal validation set. Among them, the DF model exhibited the highest discrimination compared with the other 4 models, achieving an AUC of 0.89 (95% CI 0.84-0.92). The XGBoost model followed closely with an AUC of 0.88 (95% CI 0.84-0.92). Detailed prediction performance of the 5 ML models is presented in Table 3. The DF model demonstrated the best specificity prediction for AKI (0.99, 95% CI 0.98-0.99). Compared with the other models except DF, the XGBoost model showed higher accuracy (0.92, 95% CI 0.90-0.94), PPV (0.78, 95% CI 0.68-0.86), and F₁-score (0.63, 95% CI 0.54-0.70). Calibration curves of the 5 models are depicted in Figure 4B. The results indicated that the DF model exhibited the lowest ECE value of 0.03, signifying superior calibration. The XGBoost model followed with an ECE of 0.05. Decision curve analysis of the prediction models is presented in Figure 4C. The findings revealed that the DF, XGBoost, and RF models yielded net benefits within the threshold range of 0-0.8. In the internal validation set, the DF model demonstrated the highest predictive ability for AKI in patients with CAP.

The data for the external validation set were obtained from Nanjing First Hospital and Sir Run Run Hospital affiliated with Nanjing Medical University, encompassing 3160 patients with CAP. The incidence of AKI was 20.4% (n=645) and the baseline clinical features are outlined in Table 1. Figure 5A presents the AUC of the 5 models in the external validation set. Among them, the DF model exhibited the most favorable predictive performance, achieving an AUC of 0.87 (95% CI 0.85-0.88), specificity of 0.93 (95% CI 0.93-0.94), and PPV of 0.63 (95% CI 0.59-0.67). The XGBoost model closely followed with an AUC of 0.86 (95% CI 0.85-0.88), accuracy of 0.84 (95% CI 0.83-0.85), and F₁-score of 0.65 (95% CI 0.63-0.68). Detailed performance metrics are presented in Table 4. Calibration curves of the prediction models are depicted in Figure 5B. The results demonstrated that the DF model exhibited the best calibration ability in the external validation set. Figure 5C presents the decision curve analysis curves of the 5 models, indicating that the DF, XGBoost, and RF models yielded net benefits within the threshold range of 0-0.6. In the external validation set, the DF model showcased the highest predictive ability for AKI in patients with CAP.

We used SHAP to determine the importance ranking of the 11 features, as shown in Figure 6A. The significance of SHAP features is depicted in Figure 6B, where the x-axis represents the positive or negative contribution of a single feature to the model. The y-axis displays the 11 features in descending order of importance based on the absolute value of SHAP. A higher absolute value of SHAP indicates a greater contribution of the feature to AKI risk. The results revealed that BUN, neutrophil count, CRP, white blood cell count, breathing rate, and temperature made positive contributions, implying that higher values increased the risk of AKI. Conversely, female sex, albumin, platelet count, diastolic blood pressure, and hemoglobin made negative contributions, indicating that higher values decreased the risk of AKI.

In addition to explaining the overall contribution of the model using SHAP, we used LIME to individualize the contribution of the last 11 features in the prediction process. Figure 7A illustrates the predictive process for a patient without AKI. First, the 11 patient characteristics were ranked by importance. Blue indicates the contribution to the prediction as non-AKI, orange represents the contribution as AKI, and the corresponding prediction probability is displayed on the left. The specific values of the patient’s 11 features are presented on the right. In this case, the patient had a BUN level of 5.0 mmol/L, neutrophil count of 9.52×10⁹/L, breathing rate of 20 breaths/min, temperature of 36.8°C, white blood cell count of 10.6×10⁹/L, albumin level of 24.6 g/L, CRP level of 0.5 mg/L, diastolic blood pressure level of 85 mm Hg, hemoglobin level of 110 g/L, and platelet count of 161×10⁹/L. The probability of predicting non-AKI for this patient was 1.0.

Furthermore, Figure 7B demonstrates the predictive process for a patient with AKI. With a BUN level of 19.1 mmol/L, neutrophil count of 11.5×10⁹/L, breathing rate of 30 breaths/min, temperature of 37.8°C, white blood cell count of 12.4×10⁹/L, albumin level of 34.2 g/L, CRP level of 1 mg/L, diastolic blood pressure level of 80 mm Hg, hemoglobin level of 137 g/L, and platelet count of 240×10⁹/L, the probability of predicting AKI for this patient was 0.9.

We developed a web-based prediction platform specifically designed for predicting AKI in patients with CAP (Figure 8). This platform uses the 11 selected variables as inputs to generate a risk value for predicting CAP-AKI. If the predicted risk value exceeds the predefined threshold of 74.5% set by the system, the result is considered positive, indicating that the patient falls into the high-risk category for developing CAP-AKI.

In this study, we used ML algorithms to develop an innovative prediction tool for CAP-AKI. CAP stands as a prevalent infectious disease globally, entailing considerable morbidity and mortality [11-13]. Among the array of complications linked with CAP, AKI has surfaced as a notable concern [14,15]. Our study unveils a noteworthy incidence of 15.8% for AKI among patients with CAP, indicating a substantial incidence of AKI within this population. Pinpointing the risk factors associated with AKI and crafting a predictive model are pivotal endeavors, enabling effective risk stratification and timely intervention for AKI. In this study, we identified the top 11 important predictors, including BUN, neutrophil count, respiratory rate, CRP, temperature, albumin, diastolic blood pressure, hemoglobin, platelet count, white blood cell count, and sex, as these variables can be easily collected in medical practice. In addition, our DF model showed good performance in both internal (AUC 0.89, 95% CI 0.84-0.92) and external (AUC 0.87, 95% CI 0.85-0.88) validation sets, indicating strong generalization ability.

ML techniques offer a promising path by harnessing the extensive pool of clinical data to offer insightful early-stage predictions of AKI. This approach potentially eliminates the necessity for additional AKI biomarker testing [16,17]. Hsu et al [18] constructed an AKI prediction model using 10 preprocedural indicators through an XGBoost algorithm, revealing an AUC of 0.767 on derivation and 0.761 on validation for any stage of AKI. Other researchers also found that XGBoost [19] and recurrent neural network [20] could perform well in predicting cardiac surgery–associated AKI and in-hospital AKI. Due to the distinct etiology of AKI in patients with CAP compared with other causes such as cardiac surgery–related AKI or sepsis-induced AKI, there are differences in the characteristics and factors required for model establishment. Some unique features need to be incorporated, including respiratory status, body temperature, blood pressure, and inflammatory markers, which have a significant impact on the occurrence of AKI. These parameters may differ from those in AKI models related to other etiologies. This aspect has been further elaborated in the discussion section.

In this study, we used ML algorithms to develop an innovative prediction tool for CAP-AKI. Furthermore, we determined the top 11 important predictors in the DF model as prediction model variables as these variables can be collected easily in medical activities. These features comprehensively consider the patient’s basic characteristics (such as age and sex), disease status (such as blood pressure and Scr), and potential causes (such as infections and chronic diseases), providing a well-rounded informational foundation for the model from multiple perspectives. Similarly, AUC for AKI in patients with CAP of the internal validation and external validation cohorts was shown to be 0.86 (95% CI 0.84-0.92) and 0.87 (95% CI 0.85-0.88), respectively. In addition, we constructed SHAP to provide personalized interpretation for each patient. Subsequently, an online web risk calculator model for AKI in patients with CAP was established to predict the occurrence of AKI within 7 days upon patients’ hospital admission. The median time for diagnosing AKI in our study was the third day. Although there was a certain time difference between the diagnosis time of AKI and the time of the indicators we included, our results showed that the model still exhibited good predictive performance in the validation set, and we believed that this time difference could be used as the best time for early intervention in high-risk patients who may develop AKI.

Among the 5 ML models used in this study, LR is commonly used in clinical settings. However, its performance is suboptimal when dealing with large-sample and high-dimensional data. RF exhibits good generalization ability and can prevent overfitting. XGBoost derived from RF is unaffected by multicollinearity and possesses characteristics such as flexibility and efficiency. DF is a novel ensemble-based method proposed in recent years, serving as an alternative to deep neural networks. It combines various ensemble-based methods with indivisible modules. Compared with deep neural network, DF has the advantage of achieving the optimal model with a small amount of training data, without the need for backpropagation, and has low computational costs. Zhang et al [21] found that the DF model achieved an AUC of 0.881 for predicting AKI after cardiac surgery, outperforming other ML models, including LR, XGBoost, and RF. Since its proposal, DF has been widely applied and has demonstrated good predictive performance in various fields such as radiological diagnosis of COVID-19 and emotion recognition [22,23].

In this study, we conducted a ranking analysis of the included feature variables and observed that several factors, namely BUN, neutrophil count, respiratory rate, CRP, temperature, albumin, diastolic blood pressure, hemoglobin, platelet count, white blood cell count, and sex, exerted a significant impact on the final model. Among these features, BUN emerged as the most crucial indicator for AKI. Similar to Scr, BUN serves as a reflection of renal function and carries substantial predictive weight for AKI occurrence. These findings align with the diagnostic criteria recommended by the KDIGO and are consistent with previous clinical investigations. Elevated levels of white blood cells, neutrophils, and CRP are also risk factors for AKI in patients. AKI is closely associated with renal and systemic inflammation. In the presence of infection, inflammatory factors play a critical role in combating microbial pathogens and facilitating tissue repair. White blood cells, including neutrophils, can infiltrate the injured kidneys through the circulatory system, triggering the production of cytokines, chemokines, and other inflammatory mediators, ultimately leading to renal damage [24]. CRP, as a plasma protein synthesized in the liver, serves as a nonspecific marker for systemic inflammatory response during the acute phase and is closely related to the severity of patients with CAP. Consistent with the findings of Wang et al [25] in patients with AKI after myocardial infarction, we also found that elevated CRP levels were a risk factor for AKI in the cohort of patients with CAP. Hypoalbuminemia is another risk factor for AKI in patients with CAP. Animal experiments have shown that albumin can counteract the decline in arterial vascular reactivity caused by endotoxemia, alleviate ischemia-reperfusion injury, and possess anti-inflammatory effects. Therefore, reduced albumin levels significantly reduce a patient’s resistance to infection and stress damage [26]. Similar conclusions have been reached by Bang et al [27]. The relationship between hemoglobin levels and AKI has been widely studied, with anemia being an independent risk factor for AKI [28-30]. The findings of our study provide compelling evidence that platelet levels act as a protective factor for AKI in patients with CAP. These results align with the previous investigations conducted by Zhang et al [31], thereby reinforcing the notion that a decreased platelet count is significantly associated with an elevated risk of AKI occurrence.

The prediction process of ML algorithms can be likened to a “black box” [32], where the inner workings may not be readily understandable to clinical physicians. Therefore, it is crucial for physicians to not solely rely on predictive models but also use their expertise in making appropriate diagnostic and therapeutic decisions. In this study, we used SHAP and LIME techniques to visualize the overall and individual contributions of each indicator to the model’s predictive performance, providing additional insights to clinical physicians. By quantifying the individual risks of AKI occurrence through a web-based prediction platform, we enhanced the clinical utility of the predictive model and demonstrated its significant potential in AKI risk prediction.

In this study, a comprehensive model was developed to predict the occurrence of AKI in patients with CAP by integrating multiple feature variables, resulting in robust predictive performance in both internal and external datasets. In addition, to enhance the interpretability of the best-performing DF model, visual explanations were provided using SHAP and LIME techniques.

Furthermore, a web-based prediction platform was established to augment the clinical utility of the predictive model, offering practical applications in real-world health care settings. The threshold of 74.5% was chosen based on the method of selecting the maximum F₁-score on the training set. In a real-world clinical setting, a model with low sensitivity means that a significant number of patients with AKI might not be identified. This could lead to missed opportunities for early intervention and treatment, potentially worsening patient outcomes. However, the higher PPV suggests that when the model does predict AKI, it is more likely to be correct. This reduces the burden of false positives on clinicians, who might otherwise need to follow up on many non-AKI cases, saving time and resources.

The choice of threshold reflects a preference for higher precision, which might be suitable in situations where the cost of false positives is high, such as in scenarios with limited resources or when unnecessary treatments pose risks to patients. However, the trade-off with lower sensitivity means clinicians must be aware that not all cases of AKI will be caught by the model, and they should continue to use their clinical judgment and other diagnostic tools to ensure comprehensive patient care. Remarkably, the DF model outperformed others in both the internal and external validation cohorts.

However, it is important to acknowledge the limitations of this study. First, it should be noted that this was a retrospective study, which may have introduced certain limitations in terms of the completeness, accuracy, and homogeneity of data recording compared with prospective studies. In addition, the presence of selection bias cannot be ruled out, highlighting the need for further validation through prospective research. Second, the collection of urine output data posed challenges, resulting in cases of incomplete data collection. Furthermore, the consideration of urine output standards was not incorporated in this study, which could have provided additional valuable insights. Finally, while the constructed risk prediction model shows promise, it remains uncertain whether its implementation in clinical practice would translate into tangible benefits for patients. Therefore, prospective and multicenter studies are necessary to comprehensively evaluate the effectiveness and real-world impact of the model. Acknowledging and addressing these limitations will be essential in further refining the understanding and application of the findings in future research.

In summary, this study presents a valuable early prediction tool utilizing ML techniques to identify patients with high risk of AKI among patients with CAP. The developed predictive model has the potential to aid physicians in the early identification of patients at a higher risk of AKI during their hospitalization, enabling prioritized treatment interventions and ultimately improving patient outcomes. Moving forward, our future research endeavors will focus on prospective evaluation of the effectiveness of our AKI prediction model and determining its impact on the prognosis of patients with AKI in real-world clinical practice. In addition, the model could be applied to other etiologies of AKI, such as sepsis-induced AKI, to assess its generalizability.

This study developed a ML-based prediction model for CAP-AKI, demonstrating excellent predictive performance in both internal and external validation cohorts. By integrating easily accessible clinical variables, the DF model achieved high accuracy and interpretability, offering robust support for early risk assessment and targeted intervention in patients with CAP-AKI. In the future, we hope to reduce the risk of CAP-AKI through our online prediction platform.