This study used data from 2 distinct databases: the Medical Information Mart for Intensive Care Database-IV (MIMIC-IV) [14] and the eICU Collaborative Research Database (eICU-CRD) [15]. MIMIC-IV contained critical care data for 73,181 patients admitted to the ICUs at Beth Israel Deaconess Medical Center between 2008 and 2019. As a large multicenter intensive care unit (ICU) database, eICU-CRD collected more than 200,000 ICU admissions from 335 units in 208 hospitals across the United States from 2014 to 2015. This study followed the TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) checklist (Multimedia Appendix 1).

The MIMIC-IV and eICU-CRD, which had been made publicly accessible, received ethical approval from the institutional review boards at Beth Israel Deaconess Medical Center and the Massachusetts Institute of Technology, in alignment with the principles outlined in the Declaration of Helsinki. The approval also encompassed a waiver of the need for informed consent, since all protected health information within the databases was deidentified [14,15]. Access to these databases was provided after completion of a training program in human research ethics and signing of a data use agreement with PhysioNet, earning a certification number of 27252652.

For inclusion in our study, patients were required to meet the sepsis diagnosis and initiate vasopressor treatment within 24 hours after identifying sepsis. In MIMIC-IV, sepsis was defined according to the Sepsis-3 criteria, known or suspected infection, and Sequential Organ Failure Assessment (SOFA) score ≥2 points [1]. The following 2 time points were specified to define the onset time of sepsis t_sepsis [16]: (1) t_suspicion: clinical suspicion of infection as determined by the earlier timestamp of intravenous (IV) antibiotics administration and cultures acquired within a specific timeframe. If IV antibiotics were given first, the cultures must have been obtained within 24 hours. If cultures were obtained first, then antibiotics must have been subsequently ordered within 72 hours. (2) t_SOFA: the occurrence of organ failure as identified by a 2-point increase in the SOFA score within a 24-hour period. t_sepsis is the earlier of t_suspicion and t_SOFA, as long as t_SOFA occurs no more than 24 hours before or 12 hours after t_suspicion; otherwise, the patient will not be diagnosed as sepsis. Specifically, if t_suspicion–24 hours≤t_SOFA≤t_suspicion+12 hours, then t_sepsis=min (t_suspicion, t_SOFA). In eICU-CRD, patients with sepsis were identified based on the admission diagnosis, with the sepsis identification defined as the time of ICU admission. To be as close as possible to the sepsis onset defined in eICU-CRD, only patients with sepsis onset within 24 hours of ICU admission were included in MIMIC-IV. Exclusion criteria included patients younger than 18 years, those who died within the first 24 hours of sepsis onset, or ICU stay of less than 24 hours. For patients with multiple ICU admissions, only the first admission was considered.

We hypothesized that patients with sepsis were recruited into RCTs within 24 hours of identification. Following the enrollment, all patients with sepsis were stratified based on their disease progression trajectories: “rapid death” included patients who expired within 48 hours after enrollment, “recovery” included patients who liberated from vasopressor support within 48 hours after enrollment and maintained for at least 24 hours, and “persistent ill” included patients who necessitated ongoing vasopressor support within 48 hours after enrollment.

The selection of candidate predictors incorporated both static and time-varying variables. Time-varying variables were extracted within the 24-hour period between identification and enrollment. Any outliers were identified and subsequently excluded according to the criteria defined in Multimedia Appendix 2. For time-varying variables with multiple measurements during the 24-hour duration, we included the maximum, minimum, median, and coefficient of variation (SD/mean) values for analysis. This resulted in a total of 148 features.

The MIMIC-IV data set was used for model development, and 10-fold cross-validation with 50 repeats was used for internal validation (Figure 1). The eICU-CRD was used for the retrospective external validation. A total of 12 combinations of model architectures and setups were used (Figure 1). The 4 specific architectures evaluated were gradient boosting machine (GBM) [17], neural decision forest (NDF) [18], random forest (RF), and logistic regression (LR; Multimedia Appendix 3). GBM, RF, and LR were constructed with reference to the research of Schwager et al [11] for predicting acute respiratory distress syndrome trajectories. In addition, these 3 models were commonly used in critical illness prediction tasks using electronic health record (EHR) data [19-21]. However, they were all traditional machine learning models, so we developed a deep learning model for comparison, that is the NDF, which has also been shown to be effective for disease prediction using EHR data [22].

These model architectures were used in 3 different setups, namely, the 2-way (modeling persistent ill and rapid death or recovery), 3-way nested (first predicting rapid death versus persistent ill or recovery, then classifying persistent ill or recovery), and 3-way multiclass (generating probabilities for each trajectory simultaneously) configurations. For missing data, a mean imputation method was used for NDF, RF, and LR, while GBM did not require imputation.

We used Shapley Additive Explanations [23] values to explain the best model among the 12 constructed models. In addition, we examined the performance of the lightweight model using the feature subset in 3 ways (Multimedia Appendix 3): retraining the model using only the 15 most important features (measured by Shapley Additive Explanations), using the Boruta feature selection method, or using only the maximum norepinephrine equivalence (NEE), that is, NEE (max).

To estimate the uncertainty of the model outputs, we used the conformal prediction (CP) framework built on top of the prediction algorithm (Multimedia Appendix 3). The CP is a method that can be mathematically guaranteed to make reliable predictions at a user-specified desired error rate for unknown samples that differ from the training data [13]. We specifically implemented a Mondrian CP to handle the prediction tasks with unbalanced data and worked on a class basis to ensure the desired error rate within each class. For the 3 trajectories of the patients with sepsis, the possible prediction sets are single predictions of {0}, {1}, {2}; multiple predictions of {0, 1}, {0, 2}, {1, 2}, {0, 1, 2}; and the empty prediction of {null}, where label 0 means rapid death, label 1 means persistent ill, and label 2 means recovery. We further split the development set into a training set to train the AI models and a calibration set (Figure 1) to develop the Mondrian CP and also to tune the model hyperparameters using a Bayesian optimizer [24].

Univariate analysis was performed using logistic regression and was performed by associating each variable with three trajectories: (1) rapid death versus recovery or persistent ill, (2) persistent ill versus recovery or rapid death, and (3) recovery versus rapid death or persistent ill. Coefficients and CIs were determined on the logit scale and then transformed through exponentiation to provide estimates and 95% CIs for the odds ratios.

The discrimination performance of the model was assessed using the area under the receiver operating characteristic curve (AUROC) and the area under the precision-recall curve. Calibration was performed using calibration curves and the Brier score. We mainly used the AUROC value to identify the model architecture and setup with the best performance for predicting persistent ill. Metrics such as true positive rate (TPR), positive predictive value (PPV), and the F-0.5 score (a composite of TPR and PPV that gives more weight to PPV) were also calculated. Cutoff values were chosen to maximize the F-0.5 score for the 2-way and 3-way nested setup. For the multiclass setup, the predicted outcome was obtained with the class that had the highest probability. These metrics were reported as mean (SD) for both the internal validation and external validation.

Since the conformal predictor could generate multiple or empty predictions, we could not calculate the TPR and PPV of the conformal predictor directly. To evaluate the effectiveness of the conformal predictor and assess its performance, we randomly select 1 model with the optimal architecture from the 10-fold cross-validation with 50 repeats as an example. We evaluated efficiency, which measures the proportion of all predictions that yield a single correct prediction, and the validity (error rate), representing the proportion of all predictions that did not surpass the predetermined significance level [13].

In the MIMIC-IV, 9135 patients met the inclusion and exclusion criteria. In the eICU-CRD, 3743 patients with sepsis were identified and classified as external validation cohort (Multimedia Appendix 4). The clinical characteristics of included patients are described in Table 1. In the MIMIC-IV, the median age was 68 (IQR 58-78) years, with a median SOFA score of 4 (IQR 3-5). The median NEE was 0.12 (IQR 0.07-0.25) μg/kg/minute, and the in-hospital mortality was 16.4% (n=1501). For the eICU-CRD, the median NEE was 0.15 (IQR 0.07-0.34) μg/kg/minute, and the in-hospital mortality was 22.9% (n=857). Univariate analysis was used to characterize the factors distinguishing the subpopulations of the 3 trajectories (Multimedia Appendix 5).

Patients in the MIMIC-IV exhibited the following outcomes within a 48-hour period after enrollment: 5835 (63.9%) patients experienced recovery, 2937 (32.1%) patients remained persistent ill, and 363 (4%) patients experienced rapid death. In the external validation cohort, that is, eICU-CRD, 2151 (57.5%) patients experienced recovery, 1283 (34.3%) patients remained persistent ill, and 309 (8.2%) patients experienced rapid death. In patients who recovered, the in-hospital mortality rates in the MIMIC-IV and eICU-CRD were 5.8% (n=338) and 8.5% (n=183), respectively. For patients with persistent ill, the in-hospital mortality rates in the MIMIC-IV and eICU-CRD were 27.2% (n=800) and 28.4% (n=365), respectively (Figure 2). Compared to patients with recovery or persistent ill, patients with rapid death had sustained high NEE, high heart rate, and low systolic blood pressure (SBP; Figure 3).

Of the 4 machine learning architectures in 3 model setups for predicting persistent ill on internal validation, GBM had the highest consistent performance of mean AUROC (SD) ranging from 0.806 (0.010) to 0.807 (0.010) compared to NDF ranging from 0.779 (0.013) to 0.802 (0.011), RF ranging from 0.791 (0.013) to 0.795 (0.010), and LR ranging from 0.719 (0.031) to 0.790 (0.007; Multimedia Appendix 6). Of the 3 model setups, the 3-way multiclass with GBM performed best with the mean AUROC of 0.807 (SD 0.010). In addition, the multiclass setup could also provide a probability risk score for each trajectory, with a mean AUROC (SD) of 0.906 (0.018) for rapid death and 0.843 (0.008) for recovery. In the external validation cohort, the mean AUROCs (SDs) were 0.878 (0.003) for rapid death, 0.764 (0.008) for recovery, and 0.696 (0.007) for persistent ill (Multimedia Appendix 7).

When evaluating the discrimination performance of the multiclass GBM under 4 feature subsets in identifying persistent ill patients, on the internal validation cohort, the results showed that the model achieved AUROCs of 0.664-0.807 from using NEE (max) only to using all 148 features (Figure 4A). When applied to the external validation data set, the results were AUROCs of 0.561-0.696. Figure 4B shows the calibration plot. The full metric results in different feature subsets of all 3 trajectories are presented in Multimedia Appendix 8.

The global feature importance for the multiclass GBM model using all the features is shown in Multimedia Appendix 9. The visible, individually interpretable summary of the impact of features across patients showed that oliguria, higher NEE (max), unstable oxygen saturation, higher lactate, and lower SBP were associated with a higher risk of rapid death; higher NEE (max), higher Acute Physiology Score III score, more IV fluid administrated, pulmonary infection, and lower SBP were associated with a higher risk of persistent illness. In contrast, patients with lower NEE (max), lower Acute Physiology Score III, higher SBP, more urine output, and less IV fluid administrated were more likely to recover (Figure 5A). For the example of explaining the individual prediction for different trajectories, see Figures 5B-5D.

The calibration curves of the observed prediction error were analyzed at different significance levels (1–confidence), ranging from 0% to 100% (Figure 6). When the model was evaluated using a mixed confidence approach, with a confidence level of 85% for recovery and a confidence level of 75% for rapid death and persistent ill, the model with CP reduced overall prediction errors by 27.6% (n=62) and 30.7% (n=412) in the internal and external validation cohorts, respectively, compared with the model without CP (Table 2). Specifically for the predictions in persistent ill patients, it made 69 (23.5%) errors in the internal validation cohort (compared with n=122, 41.5% errors without CP). In the external validation, it still produced significantly lower error rates compared with the model without CP (n=398, 31% vs n=689, 53.7%).

Although the number of accurately predicted persistent ill cases (efficiency) was lower compared to the conventional model without CP (n=134, 45.6% vs n=172, 58.5% in the internal cohort and n=294, 22.9% vs n=594, 46.3% in the external cohort), focusing on patients with multiple predictions of {rapid death, persistent ill} proved beneficial. Because among these predictions, 46 of 62 (74.2%) predictions were persistent ill in the internal validation cohort, and 351 of 496 (70.8%) predictions were persistent ill in the external validation cohort (Table 3).

Additionally, a web calculator was developed to enable users to visually understand the outputs of our AI model (Figure 7). This web-based calculator is internally deployed and is currently being updated at the Department of Critical Care Medicine, Zhongda Hospital. The web calculator is only accessible using a private URL by clinicians within the hospital and is connecting to the hospital’s EHR system. After a patient is identified as having sepsis and successfully collects 24-hour data after sepsis onset, the calculator would include this patient in the list. The users can then select 1 patient and view the relevant features that are calculated automatically based on data sourced from the hospital’s EHR system. After that, users can view the risk score predicted by this calculator and enter the confidence level to get a reliable prediction. The confidence level could be adjusted according to the model performance and workload, as a number of uncertain predictions can lead to an unmanageable situation.

We developed a novel AI-based model to predict patients with sepsis likely to have a long disease course. Such a model improves the homogeneity of patients with sepsis enrolled in clinical trials and ensures that a patient recruited into a trial will still be persistently ill by the time the proposed therapy can impact patient outcome. Furthermore, we used CP to estimate the uncertainty of the model outputs, which allows for a more comprehensive understanding of the model’s reliability and assists in making informed decisions based on the predicted outcomes. In addition, the developed web-based calculator may assist clinicians in comprehending the relationship between input data and predicted output. This could facilitate feedback from the clinicians and help improve the model predictions, which may be more effectively translated into medical practice for clinical decision-making.

To identify innovative pharmacological treatments for sepsis, enrolling patients who are most likely to benefit is essential. The enrichment strategy in this study differed from the prior prognostic and predictive enrichment strategies. In the Efficacy of Xuebijing Injection in Patients With Sepsis RCT, considering that the mortality rate was 74.3% in patients with a SOFA score of more than 13 in the Chinese population, Liu et al [25] used a prognostic enrichment strategy to enroll patients with sepsis with a SOFA score of 2 to 13 to evaluate the treatment effect of Xuebijing injection on 28-day mortality. However, this approach fails to consider patients with rapid recovery trajectories who may not derive significant benefits from Xuebijing injection treatment. Whether the targeted SOFA score can be effectively generalized in other trials remains uncertain. One important constraint with the predictive enrichment strategy is that different therapeutics necessitate different populations [26,27], and we lack the prior knowledge to accurately determine which patients are most likely to benefit from a specific therapy. Moreover, the limited availability of point-of-care devices for the identification of targeted patients, coupled with the lengthy measurement turnaround time, creates considerable obstacles for clinical trial enrichment [28]. These challenges are further exacerbated by the rapidly changing nature of diseases observed within clinical trials. From another perspective, we stratified patients based on the disease course of sepsis and suggested that future sepsis clinical trials should prioritize including patients with persistent ill, who would have a sufficiently long disease course to benefit from a therapy. Additionally, the current enrichment strategy does not necessarily take into account the different infection sites, pathogens, and pharmacological interventions.

Using real-time eHealth data, we developed an AI-based model for patient enrollment in sepsis clinical trials. Of the 12 machine learning models tested, a multiclass GBM model performed best and was able to provide a probability risk for each trajectory. The lightweight model using the top 15 features also showed acceptable performance and could be more suitable for future clinical applications. In addition, this model overcame the challenges of a heterogeneous patient population and tailored patient stratification to the needs of a particular drug candidate. Meanwhile, the predictive window, defined as the period between identification and enrollment, was 24 hours for the AI model, which was consistent with the previous sepsis RCTs [29]. It is reasonable to predict the disease course of patients within 24 hours after the identification of sepsis onset in future trials.

To mitigate the risk of automation bias in AI and evaluate model uncertainty, we implemented a CP framework on top of the prediction algorithm. This framework enabled the model to generate confidence measures, allowing for the monitoring of predictions and identification of uncertain outcomes based on adjustable confidence levels for human review. This is particularly crucial in different clinical scenarios where the AI system should be capable of acknowledging its limitations and saying “I don’t know” [30]. This could help to reduce prediction errors, especially in external validation, where the model performance could not be guaranteed, as well as in internal validation due to factors such as population heterogeneity and differences in clinical practice [31]. In addition, this study demonstrated that flagging the multiple (uncertain) predictions that the model could not classify for human review proved valuable, as it enabled the identification of more potentially persistent ill patients while simultaneously identifying high-risk individuals. In this case, the initial screening of patients could be achieved, allowing clinicians to avoid having to manually screen each patient for eligibility in the trial. Instead, they only need to review and assess the uncertain results of the model. As discussed by Goodman et al [32], clinicians were supposed to integrate their clinical experience with AI model outputs for clinical decision-making, thus avoiding inappropriate reliance on algorithms alone.

Our study faces some limitations. First, although this retrospective study demonstrates that the proposed AI-based enrichment strategy is effective in reducing patient heterogeneity in the design of sepsis clinical trials, we still need to include more patients to perform further prospective external validation using the developed web-based calculator. Second, the definition of sepsis onset was different between MIMIC-IV and eICU-CRD since the onset time of infection and organ failure in eICU-CRD were missing. Third, considering that our AI-based model needed a 24-hour predictive window after identifying sepsis onset, this model is not available for the mandated emergency therapies for sepsis, including the administration of antibiotics and IV fluid. The final limitation arises from the absence of criteria for balancing the trade-off between generating single-label and multiple predictions. Avoiding an excessive number of multiple (uncertain) predictions is important, as this can lead to an unmanageable situation.

Our interpretable AI-based model using clinical data accurately identifies patients with different disease courses, which can reduce the heterogeneity of patients with sepsis in future clinical trials. Additionally, we estimate the uncertainty of the model outputs and enable the performance of the model using a mixed confidence approach. Application of this model can identify more homogeneous patient populations and precision pharmacological treatments for sepsis trials.