This retrospective, single-center study was conducted at our institution following the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis guidelines. We enrolled 1267 patients who underwent LT between January 2015 and August 2022. Records were extracted using the perioperative specialist database platform (PSDP) and electronic patient record (EPR) systems. The inclusion and exclusion criteria are shown in Textbox 1.

All included recipients were formalized and registered in the China Organ Transplant Response System.

The development and temporal validation cohort datasets were created by extracting original records from the Docare System (Medical system), Hospital Information System, and Laboratory Information System, and integrating them into the PSDP platform and EPR systems. To increase ML model accuracy and applicability, we included the following variables: (1) demographic characteristics; (2) liver donor characteristics; (3) preoperative comorbidities, complications, preoperative treatment, and LT etiology; (4) preoperative laboratory test results; (5) intraoperative surgery characteristics and medications; (6) postoperative MELD scores, sequential organ failure assessment (SOFA) scores, and laboratory test results; and (7) complications and prognosis in LT recipients. All of the original data were made anonymous throughout the study.

The primary outcome was postoperative PND occurrence from surgery until discharge from the hospital. A summary of perioperative neurocognitive impairments is shown in Table S1 in Multimedia Appendix 1. The initial diagnosis criteria was the retrieval of any of the following terms from the medical records: “Delirium”, “Confusion”, “Confusional arousals”, “Clouding of consciousness”, “Soma”, “Drowsiness”, “Changes in mental status”, “Hallucinations”, “Disorientation”, “Dyscalculia”, “Haziness of spirit-mind”, “Irritability”, “Agitation”, “Inattentiveness”, “Reactive confusion”, “Somatization disorder”, “Irritability”, and “Somatoform disorders”, or equivalent terms in Chinese [4,24,25]. Next, each patient was evaluated based on the DSM-5 criteria by a designated neurologist without prior access to the patient’s records [3,26].

A comprehensive set of 137 variables was extracted for the initial analysis (Table S2 in Multimedia Appendix 1). Table S3 in Multimedia Appendix 1 provides a concise explanation of the main complications and relevant term definitions. Postoperative SOFA scores were calculated by intensive care unit (ICU) physicians immediately after surgery according to European Society of Intensive Care Medicine criteria [27] and submitted for statistical analysis.

To account for multicollinearity and confounding variables affecting the overall model fitting performance, variables that were statistically significant (P<.05) in the univariate test were subjected to stability selection (Table S4 in Multimedia Appendix 1) [28]. After 100 iterations of least absolute shrinkage and selection operator (LASSO) regression, the top 10 features with the highest selection frequencies were chosen to train the ML models. For each LASSO regression, 90% of the training set samples were randomly selected as subsamples.

The following 6 ML models were developed, and their performances were further evaluated: logistic regression (LR), multilayer perceptron classifier (MLP), extreme gradient boosting with classification trees (XGB), light gradient boosting machine (LGB), support vector machine (SVM), and random forest classifier (RF). All models were constructed using the XGB, LGB, and Scikit-learn packages.

The primary cohort dataset was randomly divided into 80% development and 20% internal validation sets. The bootstrap method was implemented 1000 times on the internal validation set to determine a 95% CI for the discrimination assessment metrics for each model: the area under the receiver operating curve (AUC), accuracy, sensitivity, specificity, and F₁-scores. Considering that ML models have multiple hyperparameters that are essential for model performance, a 5-fold cross-validation grid search method was used to optimize the parameters and AUCs (Table S5 in Multimedia Appendix 1). The Shapley additive explanations (SHAP) method was used to assess predictive feature importance and explain the ML algorithms’ predictions [29].

Because the SOFA and MELD scores have been reported as potential predictors of various post-LT complications [16,30], our study also compared the ML model’s performance against SOFA and MELD scores.

An external validation set extracted from the MIMIC-Ⅳ (version 2.2) [31] database was used to evaluate the ML model’s performance, which was authorized by the review committee of Massachusetts Institute of Technology (agreement 1.5.0). Patients who underwent LT surgery and were diagnosed with PND according to the International Classification of Diseases (9th and 10th revisions) were enrolled. Data extraction and cleaning were performed using PostgreSQL (version 15.3) and Navicate Premium (version 16) with a Structured Query Language (Figure S1 in Multimedia Appendix 1).

Data cleaning used Python (version 3.9.13) packages Pandas (version 1.4.4) and Numpy (version 1.23.5). Data analysis used the Python Scipy package (version 3.7), and SHAP (0.41.0) was used to visualize and analyze feature importance.

Data distribution was evaluated using the Kolmogorov Smirnov test. Normally distributed continuous variables are presented as mean (SD) and were compared by independent sample t tests. Non-normally distributed continuous data are presented as median (IQR) and were compared using the nonparametric equivalent (Mann Whitney test). Categorical variables are expressed as frequencies and percentages and were tested using the chi-square test or Fisher exact test. Long-term survival rates were estimated using the Kaplan Meier method. Group comparisons were conducted using the Gehan-Breslow Wilcoxon test and log-rank tests.

All tests were 2-tailed, with statistical significance set at 0.05. Before ML model training, continuous variables were normalized, dichotomous variables were coded as binary variables, and multicategory variables were coded as uniform numbers.

Variables with missing values exceeding 20% were excluded, and missing values below 20% were imputed with the median (for numeric variables) or mode (for categorical variables). The overall data distribution after imputation exhibited an acceptable level of variability.

An online calculator with a visual interface was developed to facilitate the easy input of clinical variables and to generate clear and meaningful output indicating the absolute risk in percentages.

The study protocol was approved by the Ethics Committee of the Third Affiliated Hospital of Sun Yat-sen University on July 27, 2022 (No. (2019)02-609-04) and was conducted in accordance with the Declaration of Helsinki. The requirement for informed patient consent was waived due to the study’s retrospective nature, and all data were anonymized before analysis.

The flowchart for patient recruitment is shown in Figure 1. Of the 958 patients who underwent LT, 751 patients were enrolled randomly into the development set (n=600) and internal validation set (n=151). Notably, PND occurred in 201 patients, accounting for 33.5% of the development cohort. Table 1 and Table S6 in Multimedia Appendix 1 summarizes the development set’s demographic characteristics, donor features, and perioperative variables of patients with or without post-LT PND.

Among the preoperative characteristics, American Society of Anesthesiologists classification and preoperative comorbidities such as acute respiratory distress syndrome, and laboratory results including hemoglobin, white blood cell (WBC) count, liver function, coagulation function, and serum calcium were significantly different between patients with and without postoperative PND (P<.01, Table S6 in Multimedia Appendix 1). Specifically, individuals diagnosed with post-LT PND exhibited a notably elevated prevalence of preoperative cover hepatic encephalopathy (CHE; 45.27% vs 8.77%, P<.001) and hypercalcemia (7.57% vs 1.36%, P<.001). Furthermore, patients with post-LT PND had higher Child Pugh and MELD scores (P<.001), longer preoperative ICU stays, increased continuous blood purification, increased plasma exchange, longer mechanical ventilation, and higher tracheal intubation (all P<.001, Table S6 in Multimedia Appendix 1).

Regarding intraoperative characteristics, patients with post-LT PND had longer anesthesia durations; increased sodium bicarbonate levels, red blood cell counts, plasma levels, and levels of cryoprecipitate transfusion; increased estimated blood loss (EBL); and reduced urine output (all P<.001, Table S6 in Multimedia Appendix 1). Differences in intraoperative medications between the 2 groups were not significant, except for recombinant activated factor VII (P<.001). Interestingly, our results showed no association between day or night surgery and the incidence of PND (P=.44, Table S6 in Multimedia Appendix 1).

For the postoperative characteristics, patients with post-LT PND showed significantly higher levels of aspartate aminotransferase (AST), total bilirubin, blood urea nitrogen, prothrombin time (PT), international normalized ratio, hypersensitive C-reactive protein (hsCRP), procalcitonin, and serum calcium, as well as lower levels of hemoglobin, hematocrit, WBC, platelet (PLT), gamma-glutamyltransferase, albumin, and serum osmolality (all P<.05, Table S6 in Multimedia Appendix 1).

The frequency of LASSO algorithm selection for each variable is shown in detail in Figure S2 in Multimedia Appendix 1. The top 10 features chosen as predictors for ML model development were preoperative CHE, PLT, PT, estimated glomerular filtration rate (eGFR), Ca²⁺, MELD score, intraoperative EBL, postoperative SOFA score, hsCRP, and AST.

The performance of the 6 ML models is shown in Figure 2. The LR model achieved the highest AUC (0.799, 95% CI 0.709-0.877) with acceptable accuracy (0.722, 95% CI 0.642-0.795), sensitivity (0.714, 95% CI 0.575-0.833), and specificity (0.73, 95% CI 0.639-0.811) compared with the other 5 models.

The SOFA (AUC=0.459, 95% CI 0.365-0.555), preoperative MELD (AUC=0.672, 95% CI 0.581-0.768), and postoperative MELD scores (AUC=0.679, 95% CI 0.587-0.772) had significantly lower AUCs than the LR model in the internal validation set (Figure 3A).

The SHAP summary plot (Figures 3B and 3C) illustrates the correlation between the feature value magnitudes in the LR model. Both SHAP plots revealed that the presence of CHE, lower preoperative PLT, higher postoperative SOFA score, higher postoperative hsCRP, and higher preoperative PT were associated with a higher SHAP value output in the LR model, indicating a heightened likelihood of post-LT PND and forming the top 5 effective variables.

Three correctly classified examples (eg, patients 48, 80, and 122) are presented in Figure S3 in Multimedia Appendix 1, showing the SHAP decision and force plots.

A comparison of the main demographic characteristics and key predictive variables between the development and validation sets is shown in Table S7 in Multimedia Appendix 1, and the incidence rates of post-LT PND in the temporal and MIMIC-Ⅳ external validation were 27.1%, and 20.3%, respectively. The LR model exhibited a comparable performance in the temporal external validation set (AUC=0.826, 95% CI 0.765-0.887) (Figure 3D). Surprisingly, the LR model also provided acceptable predictions for the MIMIC-Ⅳ dataset (Figure 3D, AUC=0.72, 95% CI 0.606-0.829). Figure 3E summarizes the main performance metrics of the LR model.

Compared with patients without post-LT PND, patients with PND were more likely to experience perioperative complications (Table S8 in Multimedia Appendix 1), including higher incidences of sepsis (51.63% vs 21.55%, P<.001), pneumonia (75.56% vs 65.46%, P<.05), acute kidney injury (69.5% vs 39.75%, P<.001), and hemodialysis (51.35% vs 12.81%, P<.001). Furthermore, patients with post-LT PND had higher hospitalization costs (CNY 377,801.69 [US $51,566.83], SD 177,855.53 [US $24,275.82] vs CNY 277,018.95 [US $37,810.82], SD 92,779.91 [US $12,663.70]; P<.001), prolonged postoperative stays (25 {18} vs 21 {11} days, P<.001), longer postoperative ICU stay (113 {114} vs 65 {48.5} hours, P<.001), and a markedly higher in-hospital mortality rate (12.44% vs 2.51%, P<.001).

Further survival analysis (Figure 4) was conducted to assess patient prognosis. The PND group exhibited significantly lower survival rates at 30 days (91.5% vs 98.2%, P<.001), 3 months (90.3% vs 97.1%, P<.001), 6 months (89.1% vs 96.1%, P<.001), and 12 months (87.9% vs 92.6%, P=.02).

Given the accessibility of the 10 predictive features, we constructed a visually oriented online calculator to facilitate clinical decision making. The perioperative information of 2 typical patients was entered into the online calculator: patient 48 had a positive final predicted probability of PND occurrence (probability: 96%), and patient 122 had a negative final predicted probability of PND occurrence (probability: 17%; Figure 5). The online calculator is freely accessible at the hospital website.

Our retrospective study assessed 6 different ML algorithms to predict post-LT PND, using 10 readily available clinical parameters. We found that post-LT PND incidence was 33.5%. The 10 predictive features significantly associated with PND included preoperative CHE, PLT, PT, eGFR, Ca²⁺, MELD score, intraoperative EBL and postoperative SOFA score, hsCRP, and AST. The LR model demonstrated superior performance, with high AUC, accuracy, sensitivity, and specificity, surpassing traditional SOFA and MELD scores in predicting post-LT PND and performed acceptably in the rigorous temporal and MIMIC-Ⅳ external validations.

This study aids clinicians in detecting postoperative cognitive changes in LT recipients. Patients with PND typically faced more perioperative complications, higher hospitalization costs, and prolonged hospital and ICU stays, consistent with previous studies [4,23]. Hepatic encephalopathy has been reported as an independent risk factor for postoperative neurocognitive disorders [32]. To ensure cognitive assessment accuracy, we excluded patients with overt hepatic encephalopathy according to the spectrum of neurocognitive impairment in cirrhosis criteria [33]. CHE emerged as a significant predictor in our model analysis. Both oxidative stress and neuroinflammation have been implicated in POD pathophysiology [10,34]. A recent systematic review also links increased perioperative CRP levels to a high delirium risk [35], supporting our inclusion of hsCRP as a predictor. Calcium ions (Ca²⁺) are important cell signaling molecules, and previous studies reported a positive correlation between Ca²⁺ concentration and neuronal apoptosis extent in vitro [36], consistent with our results. Furthermore, the model identified PLT as an unconventional indicator of PND, showcasing ML’s ability to highlight nontraditional risk factors. This discovery is partly supported by Eyer et al [37] suggesting a relationship between lower PLT and delirium tremens.

Our study used preoperative, intraoperative, and postoperative data (SOFA scores, hsCRP, and AST levels) to develop the LR model. Earlier studies have revealed that multiple postoperative factors were also risk factors for PND [11,12]. The postoperative variables included in this study were predominantly assessed upon initial admission to the ICU. Stability selection analysis revealed a positive correlation between elevated postoperative SOFA scores, hsCRP levels, and AST levels, and an increased likelihood of post-LT PND. This highlights the predictive value of these commonly observed postoperative variables for PND.

Our results suggest that LR outperforms other ML models in predicting post-LT PND, which is not surprising. A recent systematic review showed no performance superiority of other ML models over LR in predicting clinical complications [38]. Wiredu et al [35] also found that compared to ML algorithms, LR had the highest AUC when predicting sex-specific hip fractures. Song et al [4] developed an LR model to predict POD in older adult patients, achieving the highest AUC compared with other models. Given the evident linear relationships among the top 10 features, the LR may be more appropriate for capturing distribution patterns. In contrast to other algorithms, LR performs well on nonoversized and high-dimensional datasets, exhibits computational efficiency, and imposes lower dataset requirements.

As demonstrated by the example of prediction cases (Figure 5), we successfully developed a predictive model for post-LT PND, with its primary advantage in its reliable predictive performance, validated using 2 external datasets. The importance of early detection and prevention of PND in patients undergoing cardiac surgery or transplantation is clearly emphasized in current international guidelines [39]. However, the implementation of preventive measures is often challenged by limited resources [40], especially in cases where the shortage of liver donors persists. On accurate identification by the LR model, patients at high risk for post-LT POD could be referred to enhanced LT perioperative management strategies, such as individualized pharmacological or nonpharmacological comprehensive multicomponent interventions, according to the 10 commonly accessible predictive parameters filtered by the ML algorithm.

However, this study had several limitations. First, it was a single-center retrospective study, meaning the Confusion Assessment Method (CAM) or the associated CAM-ICU and 3D-CAM were inappropriate for our database. Instead, patients with PND were identified from medical records according to the DSM-5 criteria [3,6,26]. Second, as a real-world study, researchers can only infer precise risk factors based on the data available, and inhomogeneous confounding among the datasets could affect the study conclusions [41]. While our online decision tool has the potential to aid surgeons and anesthesiologists in clinical decision making, the causes and underlying mechanisms of PND remain subjects of intense debate, necessitating further research.

This study successfully develops a real-time and easily accessible parameter requiring LR-based PND prediction algorithm for post-LT settings. The LR model outperformed the other five models owing to its enhanced model performance and interpretability. The optimal use of our freely accessible online predictor would enable timely and convenient risk stratification, enhanced perioperative management strategies, and comprehensive multicomponent interventions.