The study was carried out following the 2020 PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) statement (Multimedia Appendix 1) [15] and was preregistered with PROSPERO (International Prospective Register of Systematic Reviews, ID: CRD42024549357) [16].

The inclusion criteria for this study included cohort, case-control, and cross-sectional studies. The study population consisted of individuals diagnosed with ARDS according to internationally recognized guidelines at the time of the study with no restrictions on the underlying causes of ARDS. As this study aimed to systematically review the effectiveness of ML-based tools in predicting ARDS mortality, the observation group included patients with ARDS who died during the follow-up period, regardless of the specific causes of death. The primary outcome measure was the concordance index (C-index), which reflects the overall accuracy of the predictive models. In addition, the frequency of variables used in model development was considered a secondary outcome measure.

Studies were excluded if they were expert opinions, meta-analyses, guidelines, reviews, or unpublished conference abstracts. Research that did not involve the construction of a predictive model and instead examined risk factors was also excluded. Furthermore, studies lacking outcome measures necessary for assessing model accuracy, such as accuracy, C-statistic, C-index, receiver operating characteristic (ROC) curve, sensitivity (SEN), specificity (SPE), F₁-score, recall, precision, confusion matrix, calibration curve, or diagnostic 2×2 tables, were not considered. In addition, studies with an insufficient sample size, defined as fewer than 20 participants, were excluded from this review. Finally, papers that were not published in English were excluded.

Up until April 27, 2024, we performed a thorough search across PubMed, Web of Science, the Cochrane Library, and Embase. Both MeSH (Medical Subject Headings) and free-text terms were used for searching with no regard to publication year or region. To identify and acquire pertinent material, the references of the included research were also retrospectively searched. Details are provided in Table S1 in Multimedia Appendix 2.

After being imported into EndNote (version 20; Clarivate), retrieved studies were screened to remove duplicates. Following this, the titles and abstracts of the retrieved publications were evaluated to identify preliminarily eligible original studies. Then, full texts of these studies were downloaded to screen qualified studies. For data extraction, we formulated a standardized electronic data extraction form to capture essential information, including title, DOI, first author, publication year, country of the authors, study type, patient source, disease background, and follow-up duration. The data extraction and literature screening procedures were completed independently by 2 researchers. Cross-checking was completed to ensure consistency. In case of discrepancies, a third researcher was consulted to discuss and determine whether the study should be included. The Cohen κ coefficient was used to assess the consistency of the 2 researchers’ assessments. The Cohen κ coefficient for the final screening results was 0.823, indicating a high degree of agreement between their assessments.

To assess the risk of bias in the included original studies, the Prediction Model Risk of Bias Assessment Tool (PROBAST) was used. PROBAST encompasses four domains with multiple questions: participants, predictors, outcomes, and statistical analysis, which collectively reflect the overall risk of bias and overall applicability. Each of these domains contains specific questions, with 3 possible responses for each question: “Yes” or “Probably yes” (low risk of bias), “No” or “Probably no” (high risk of bias), and “No information.” If all domains were judged as low risk, the total risk of bias was assessed as low, if at least 1 domain was judged as high risk, the total risk was rated as high. Using PROBAST, 2 researchers independently appraised the risk of bias. After completion, they cross-checked the assessments, and a third researcher was involved to resolve discrepancies and reach a final consensus in cases of disagreement.

C-index, an indicator of the overall accuracy of ML models, was meta-analyzed. For original studies lacking a 95% CI or SE for the C-index, the SE was estimated based on the methods described by Debray et al [17]. For the meta-analysis of the C-index, a random-effects model was prioritized due to the varied parameters and variables in ML models. Furthermore, a bivariate mixed-effects model was applied to analyze SEN and SPE. SEN and SPE were then meta-analyzed based on diagnostic 2×2 tables. However, as most original studies did not report diagnostic 2×2 tables, we calculated these metrics using SEN, SPE, and precision based on the number of cases. All related meta-analyses were done in R (version 4.2.0, R Development Core Team).

Our study was a systematic review that used nonidentifiable, secondary data from published studies. According to institutional policies, no ethics review was required.

Overall, the search yielded 18,849 papers, including 2434 from PubMed, 6091 from Web of Science, 3957 from Cochrane Library, and 6367 from Embase. After removing 3542 duplicates and excluding studies irrelevant to the topic per title and abstract, 32 papers were identified for further review. These preliminarily eligible studies were downloaded and thoroughly read, resulting in 21 original studies [18-38] involving a population of 31,291 patients with ARDS being included (PRISMA flow diagram is presented in Figure 1).

Among the 21 studies included a total of 31,291 patients with ARDS were analyzed. All 21 studies were cohort studies with 6 [18,20,24,32,33,37] being prospective cohort studies and 15 [19,21-23,25-31,34-36,38] being retrospective cohort studies. Of these, 7 [19,24,28,31,32,34,37] studies were conducted in single-center settings, 8 [18,20-22,27,33,36,38] were multicenter studies, and 7 [23,25-27,29,30,35] used data from registry databases, primarily the Medical Information Mart for Intensive Care III (MIMIC-III) database. The majority of the multicenter research was carried out in developed areas such as Europe and North America. A total of 11 studies [18,19,21,22,27,30,31,33,34,36,38] provided detailed information on the disease background, 4 studies [19,21,36,38] focused on patients with ARDS receiving extracorporeal membrane oxygenation therapy, and 3 studies [22,30,33] focused on those undergoing mechanical ventilation. In terms of mortality outcomes, 18 studies [18-23,25,27-35,37,38] focused on in-hospital mortality, while 6 studies [19,23-26,36] examined out-of-hospital mortality. Out of the 21 studies, 12 [18,21,22,25,26,28-30,33-35,38] included independent validation sets, 6 [21,22,25,29,30,38] of these used external validation sets, while 10 [18,21,26,28-30,33-35,38] used internally validated datasets based on random sampling. The studies evaluated 5 different models with a primary focus on logistic regression. Other models included multilayer perceptron, random forest, Extreme Gradient Boosting (XGBoost), and radial basis function. In total, 9 studies [18,19,24,25,27,30,32,33,38] compared ML models with conventional scoring tools, 6 studies [18,19,25,27,30,38] provided SOFA scores, 4 [25,27,30,38] provided SAPS II scores, 1 [19] provided APACHE II scores, and 1 [30] provided APACHE IV scores (Table S2 in Multimedia Appendix 3).

We assessed the risk of bias for the 26 ML models included in our review. The results indicated that 6 models were based on retrospective cohort studies, resulting in a high risk of bias concerning study participants. As all studies were cohort studies, there was no high risk of bias regarding predictors. Since predicted mortality was the outcome, no complex assessment was needed for outcome evaluation, suggesting a low risk of bias in this domain. In terms of statistical analysis, 16 models did not meet the criterion that the number of death cases in the training set should be 20 times the number of modeling variables, nor did they meet the criterion that the number of cases in the validation set (Events Per Variable >20) should exceed 100. These models were therefore rated as having a high risk of bias. In addition, 6 models had inappropriate methods for handling missing values, indicating a high risk of bias. Similarly, 6 models also used unsuitable methods for model fitting. The risk of bias assessment results is detailed in Figure 2. In addition, to assess potential publication bias, we conducted Egger tests for various scores, training sets, and validation sets, respectively. The results of the publication bias and Egger tests are presented in Figures S1-S6 in Multimedia Appendix 4.

A total of 26 models were constructed in the training sets, yielding a pooled C-index of 0.84 (95% CI 0.81-0.86; Figure 3 [18,20-27,29-38]). Forest plot summarizing the predictive performance of various models for in-hospital, 30-day, 90-day, and 1-year mortality using the C-index (95% CI). The overall pooled estimate indicates strong predictive ability with substantial heterogeneity across studies [18,20-38]. Among these, 23 models predicted in-hospital mortality, yielding a pooled area under the ROC curve (ROC-AUC) of 0.83 (95% CI 0.81-0.86). In the validation sets, 26 models were also constructed, yielding a pooled C-index of 0.81 (95% CI 0.78-0.84; Figure 4 [18,21,22,25,26,28-30,33-36]). Forest plot displaying the C-index (95% CI) of different predictive models for in-hospital, 30-day, and 1-year mortality. The pooled analysis suggests moderate to high predictive performance with heterogeneity present among the included studies [18,21,22,25,26,28-30,33-36]. Among these, 24 models predicted in-hospital mortality, yielding a pooled ROC-AUC of 0.80 (95% CI 0.77-0.84). A total of 13 cohorts validated the SOFA score, resulting in a pooled ROC-AUC of 0.64 (95% CI 0.62-0.67) and 11 cohorts validated the Simplified Acute Physiology Score II (SAPS-II) score, showing a pooled ROC-AUC of 0.70 (95% CI 0.66-0.74; Figure 5 [18,19,24,25,27,30,32,33,38]). Forest plot comparing the C-index (95% CI) of various severity scoring systems, including APACHE, SAPS-II, SOFA, and other models, for mortality prediction. The overall predictive performance varies across different scoring systems, with notable heterogeneity in some subgroups [18,19,24,25,27,30,32,33,36].

In the newly constructed ML models, we extracted all predictors. Key predictors included age, lactate level, oxygenation index, creatinine level, platelet count, albumin, white blood cell count, body temperature, and immune function impairment (details are provided in Table S3 in Multimedia Appendix 5).

To assess the robustness of the findings, a SEN analysis was conducted. We assessed whether individual studies significantly altered the overall pooled C-index by sequentially excluding each study. The results remained consistent, suggesting that no single study disproportionately affected the conclusions. We stratified studies based on key factors such as model type, dataset size, and outcome definitions to determine whether heterogeneity was driven by specific subgroups. The findings indicated that some model categories exhibited lower heterogeneity than others. The results of the SEN analysis are uploaded as Figures S7-S9 in Multimedia Appendix 4.

This meta-analysis found that while the SOFA and SAPS-II scores exhibited certain predictive performance for ARDS mortality, their overall effectiveness remains a concern. We acknowledge that traditional scoring systems offer well-established, interpretable tools for clinical decision-making with a long history of validation in various critical care settings. However, they may have limitations in capturing complex, nonlinear relationships within heterogeneous patient populations. For instance, SOFA and SAPS-II scores rely on predefined variables and thresholds, which may not adequately adapt to dynamic patient conditions or account for interactions between multiple risk factors. These limitations can lead to reduced accuracy in certain subgroups, such as patients with atypical presentations or those with multimorbidity.

Conversely, ML models excel in identifying complex patterns in high-dimensional data and can provide more individualized risk assessments. They may offer superior predictive performance in cases where traditional models fail to fully capture patient heterogeneity, such as sepsis, which involves rapidly evolving pathophysiological changes. In addition, ML models can complement traditional scoring systems by integrating additional predictive features and allowing real-time updates based on new clinical data. On this basis, the new ML models analyzed in this study demonstrated superior predictive performance with a pooled C-index of 0.81 (95% CI 0.78-0.84) in the validation sets and that of 0.80 (95% CI 0.77-0.84) for in-hospital mortality.

Previous studies have explored early prediction methods for ARDS mortality risk, but they mainly focused on identifying risk factors. For example, in a study by Jayasimhan et al [39], dead space ventilation was identified as an independent risk factor for ARDS mortality. Similarly, Terpstra et al [40] confirmed in their review that a reduced protein C level is associated with increased ARDS incidence and mortality. Other research has shown that an elevated level of angiopoietin-2 can independently predict mortality risk in patients with ARDS [41]. However, in clinical practice, mortality risk assessment based solely on individual risk factors remains vague and poses significant challenges.

Researchers have also explored other predictive methods in hopes of achieving more accurate mortality predictions for ARDS. In a meta-analysis of cardiac biomarkers in ARDS, it was confirmed that several cardiac biomarkers, including N-terminal pro-B-type natriuretic peptide and B-type natriuretic peptide, were associated with higher mortalities [42]. Zeng et al [43] found that angiopoietin-2 had certain prognostic value for patients with ARDS, with a ROC-AUC of 0.83, SEN of 0.69, and SPE of 0.81. However, the study exhibited considerable heterogeneity in prognostic analysis, and angiopoietin-2 was only a single predictor. Some researchers have attempted to predict short-term mortality in patients with ARDS using computed tomography. Unfortunately, the results showed SEN of 0.62 (95% CI 0.30-0.88) and SPE of 0.76 (95% CI 0.57-0.89) [44], indicating significant limitations in using computed tomography for short-term mortality prediction in patients with ARDS. Building on this foundation, our study summarized ML models constructed from commonly used and interpretable risk factors and biomarkers, which demonstrated superior predictive performance for ARDS mortality of these models. Therefore, future research could focus on developing simple and widely applicable scoring tools using ML methods and these multivariable clinical factors.

Rashid et al [45] performed a systematic review of the application of ML in ARDS. However, their study included only 2 original studies on mortality prediction, and these studies did not reach conclusions regarding the accuracy of mortality prediction. Tran et al [46] included 12 studies that explored the management, prediction, and classification of ARDS based on ML methods. However, they did not address the applicability or provide definitive conclusions regarding these approaches. In our study, we presented a more comprehensive evidence summary and concluded that new ML models have favorable predictive performance for ARDS mortality, with a pooled C-index of up to 0.80 for in-hospital mortality.

This study provides evidence-based support for ML-based models for the prediction of mortality risk in ARDS. However, limitations that should be considered are as follows. First, even though we systematically searched several databases, only a limited number of studies met the inclusion criteria. Second, in constructing ML models, diverse modeling methods were used; however, due to limited evidence, this study analyzed only a small subset of studies on certain ML methods, which restricted the depth of our discussion. Third, external validation is crucial for assessing model performance across different populations, reducing the risk of overfitting, and ensuring broader clinical applicability. However, challenges such as data availability, patient heterogeneity, and institutional differences often hinder external validation efforts in this field. The studies included in this meta-analysis primarily relied on internal validation using random samples, which may limit the interpretability of the results. Future studies should incorporate multicenter datasets with diverse demographic and clinical characteristics, and prospective validation using external datasets from different institutions is necessary to assess real-world performance. In addition, to improve clinical adoption, future models should incorporate interpretable techniques, such as Shapley additive Explanation values or attention mechanisms, to improve transparency. Lastly, the causes of ARDS are highly varied and come from a variety of illness backgrounds. However, the limited evidence included in this study prevents a more in-depth exploration of the different etiological backgrounds of ARDS. Regarding the representativeness of the included studies, we acknowledge that differences in study design, patient populations, and geographic distribution may affect the generalizability of the findings. To improve reproducibility, we recommend that future studies adopt standardized reporting guidelines, such as TRIPOD+AI (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis+Artificial Intelligence), to ensure comprehensive and transparent documentation of data preprocessing, model development, and validation procedures. In addition, promoting data and code sharing could facilitate independent verification of findings.

ML-based models for predicting ARDS mortality risk demonstrated relatively strong predictive performance, surpassing that of commonly used individual scoring tools. However, our findings are based on limited evidence, which imposes certain limitations on its value for guiding clinical practice. Therefore, clinicians should exercise caution in interpreting and applying these findings. Future studies are expected to further develop or update efficient, simplified, and clinically applicable ML-based ARDS mortality risk prediction tools. Such tools could facilitate early identification of ARDS mortality risk and aid in formulating individualized, personalized, risk-based prevention strategies.