The review was performed according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines and flowchart [19,20] and the PRISMA diagnostic test accuracy checklist (Multimedia Appendix 1) [21]. It was registered in the PROSPERO database (registration number: CRD42022329252).

We searched PubMed, Scopus, LitCovid, Embase (using the OVID platform), and the Cochrane Library (CENTRAL) from December 1, 2019, to April 30, 2022. The search included terms related to COVID-19, DL, and disease severity (Textbox S1 in Multimedia Appendix 2). In addition, another person independently collected literature through citation searches. After removing duplicates, 2 reviewers (CW and YT) independently performed an initial screening of titles and abstracts using Endnote X9 (Clarivate) software and then independently assessed articles against the inclusion criteria using Zotero software (Corporation for Digital Scholarship). Disagreements were resolved by discussion and, where necessary, by third-party adjudication.

The inclusion criteria were (1) evaluating the assessment or predictive value of DL algorithms on disease severity in patients with COVID-19; (2) disclosing the code of the DL algorithm or detailing the parameters used by the model, such as training epochs, learning rate, batch, optimizer, validation strategy, and so forth; (3) reconstructing a 2×2 confusion matrix from sensitivity, specificity, positive predictive value, and negative predictive value; and (4) peer-reviewed articles. Reviews, protocols, and editorials were excluded. Studies that did not clearly indicate the source of the patient data sets were also excluded.

The QUADAS-2 (Quality Assessment of Diagnostic Accuracy Studies 2) criteria assessed the risk of bias in 4 domains: patient selection, index test, reference standard, and flow and timing. However, QUADAS-2 cannot be evaluated against predictive models for diagnosis or prognosis [22], and to refine this, we further introduced the PROBAST (Prediction Model Risk of Bias Assessment Tool) [23], which is well suited to address DL predictive models for binary outcomes [24]. Furthermore, PROBAST assessed the risk of bias in 4 other domains: participants, predictors, outcomes, and analysis.

Statistical analysis was performed with STATA (version 17.0) using the MIDAS module [25] and the METAPROP module [26]. Postestimation procedures for model diagnostics and empirical Bayesian predictions were used to assess heterogeneity using the I² statistic. The following metrics were used: 0%-40% (not important heterogeneity), 30%-60% (moderate heterogeneity), 50%-90% (substantial heterogeneity), and 75%-100% (considerable heterogeneity) [27]. Deek funnel plots were tested for publication bias using an asymmetry test. If P<.10, publication bias may be present. Using bivariate mixed-effects logistic regression modeling [25], forest plots were used to compare the sensitivity and the specificity of DL models for assessing and predicting disease severity in patients with COVID-19. Summary receiver operating characteristic (SROC) curves were adopted to assess overall diagnostic accuracy. The Fagan nomogram was used to explore the relationship between pretest probability, likelihood ratio (LR), and posttest probability. LR dot plots, divided into 4 quadrants based on the strength of the evidence threshold, were used to determine the exclusion and confirmation of the DL model. Finally, subgroup analyses were performed to examine whether the estimated sensitivity, specificity, and associated I² (when each subgroup included 4 or more studies) differed by a number of moderators. Details are provided in Textbox S2 in Multimedia Appendix 2.

A total of 1154 titles and abstracts were identified in the initial search. According to this study’s inclusion and exclusion criteria, 1044 articles were excluded. In addition, 110 studies were reviewed for full text, of which 23 met all inclusion criteria (Figure 1).

The PRISMA 2020 flowchart for new systematic reviews included searches of databases, registers, and other sources [20].

All studies were retrospective and used completely different data sources. Eleven of these studies classified the stage as severe or critical according to the guidelines for diagnosis and treatment of COVID-19 infection from the National Health Commission of the People’s Republic of China [28-38]. However, except for the study in which disease was determined by scoring the image parameters [39,40], all other studies defined severe patients as having at least one of the following criteria: respiratory rate ≥30 breaths/min, oxygen saturation ≤93% at rest, PaO₂/FiO₂ ≤300 mmHg, significant progression of pulmonary lesions (>50%) within 24-48 h, mechanical ventilation, intensive medical care, or shock. Details of the criteria for severe patients in different studies, study type, and the design characteristics of the DL model are provided in Tables S1 and S2 in Multimedia Appendix 2. Table 1 summarizes the characteristics of the included studies and the diagnostic test accuracy of the DL prediction model.

We performed the subgroup analyses in 6 areas, including data partition (internal test or external test), data sources (single benchmark or multiple benchmark), training method (pretrained or customized), DL model networks (ResNet or other networks), input parameters (image parameters only or clinical and image parameters), and image (computed tomography [CT] or x-ray), to effectively understand how the different 6 types affected the performance of the algorithm for COVID-19 assessment and prediction.

In sensitivity, from univariable meta-regression and subgroup analyses (Figure S2 in Multimedia Appendix 2), we can learn that all domains influenced the heterogeneities of sensitivity for assessing and predicting disease severity, but none of the 6 influenced the DL model for assessing and predicting COVID-19 severity (Table 2), as their heterogeneities were very low (I²=0.00%).

In terms of data partitioning, the specificity of the internal test and external test data sets for assessing disease severity was 0.98 and 0.85, respectively, with significant heterogeneity between groups (P<.001). On the other hand, subgroups based on sources (P=.001), training method (P=.01), input parameter (P=.02), or image (P<.001) may have intergroup heterogeneity in the specificity of prediction. Among them, the specificity of 0.90 for a single source was higher than that of 0.80 for a multicenter. Furthermore, the customized training method achieves a specificity of 0.87, while the pretraining method achieves only 0.80. Additionally, the specificity of the parameter that included both clinical and image data was 0.83, while the parameter that included only image data was 0.73. Finally, the specificity of the DL model using x-ray was 0.78, which was significantly lower than the specificity of the model using CT, which was 0.84. Detailed results of the subgroup analyses are shown in Table 2, and corresponding plots are presented in Figure S3 in Multimedia Appendix 2.

Among the DL models included in the systematic review, CT was used more frequently than x-ray: CT was used in 10 of the DL models assessing COVID-19 severity and in 10 of the models predicting severity. However, there is no significant difference in their impact on model performance.

After evaluating sensitivity, specificity, and LR together [53], we found that DL achieved higher sensitivity and specificity in assessing the severity of COVID-19 compared to using CT [54] or neutrophil-lymphocyte ratio (NLR) alone [55]. However, DL models for longitudinal prediction of disease severity failed to exclude and confirm patients. Although the DL model was significantly superior to thrombocytopenia in predicting disease progression [56], the results with NLR resembled the ones obtained using DL [57,58].

The parameters used in the DL model should be derived from predictor variables that are known predictors in the scientific literature, thus limiting overfitting [59]. However, only 4 of the 23 articles used this approach to select predictor variables [41,44,45,47]. Of the remaining articles, 10 adopted univariate variables [29-32,37,39,40,42,43,46], and 9 used variables with significant levels in clinical analyses [28,33-36,38,48-50]. However, univariate variables or variables with significant levels in clinical analyses may not be suitable as candidate predictors [60]. We specified a list of candidate predictors (Table S3 in Multimedia Appendix 2), which were summarized in a systematic literature review of prognostic factors affecting COVID-19 prognosis. However, the number of predictors needs to be determined by the sample size [61]. Too many predictor variables may, on the one hand, prevent the model from providing valid estimates in new patients [62] and may include variables that are not relevant to the outcome and lead to test bias [62,63]. This unfavorable situation occurred in 5 of our included studies [33,38,44,49,50].

Model exploitation requires both a training set (ie, a developmental data set) and a validation set (ie, an internal validation data set) [64]. Once the predictive model is complete, an external test set (ie, an external validation data set) is strongly recommended to evaluate the performance of the model [65], but only 7 articles have done so [28,29,33,34,37,38,45]. The internal test set generated by temporal partitioning (ie, the temporal validation data set) is considered effective as an intermediate between the validation set and the external test set [18]. This approach was used in 3 of the 18 studies that used internal test sets [45-47]. However, the remaining 15 generated internal test sets with random splitting [30-32,34-36,39-44,48-50], which was considered inefficient [64].

The DL prediction model has relatively low heterogeneity with respect to sensitivity but considerable heterogeneity with respect to specificity. As a result of the sensitivity analysis (Table 2), for specificity, the heterogeneity in assessment comes mainly from the data partitioning, whereas the heterogeneity in prediction comes from 5 aspects: data partitioning, data sources, training method, DL model networks, and image. However, there is no significant difference in these 5 aspects, which may be related to the low performance of the vertical prediction model.

The specificity of the external test data set was significantly lower than that of the internal test data set, suggesting that the study warrants external validation [18]. Although there may be intergroup heterogeneity in the specificity of COVID-19 severity prediction based on subgroups of sources, training methods, input parameters, or images, they are all unevenly distributed within their subgroups. Therefore, the impact of these 4 aspects on the specificity of DL prediction models needs to be further investigated. In DL model networks, Komolafe et al [53] found a difference between ResNet and other network models in detecting COVID-19, whereas our study found no significant difference in sensitivity or specificity between ResNet and other network architectures in diagnosing and predicting the severity of COVID-19 (Table 2). This result suggests that, unlike in disease detection, changing the network architecture alone may have little significant impact on DL performance and that factors such as subgroups of sources, training methods, input parameters, and images need to be taken into account.

The study has several limitations. First, all included studies were retrospective, which may introduce bias due to missing information and unavailable confounders [66]. Second, all of these studies lacked large-scale clinical data. Third, although the effect of 6 aspects on the DL model to assess and predict severity was investigated, no further analysis of specific clinical factors, such as NLR and disease process spectrum, was performed [18]. Finally, only 7 articles used external tests [28,29,33,34,37,38,45], and no studies explicitly cited the TRIPOD (Transparent Reporting of a Multivariable Prediction Model of Individual Prognosis or Diagnosis) [64].

The meta-analysis showed a remarkably high performance of the DL model for assessing COVID-19 severity and good predictive values for disease severity. However, high-quality studies are lacking. We hope that more researchers will take advantage of the upcoming TRIPOD-AI (Transparent Reporting of a Multivariable Prediction Model of Individual Prognosis or Diagnosis–Artificial Intelligence) to standardize their studies on DL or machine learning prediction models [67]. Significantly, the predictive performance of DL for COVID-19 severity leaves much to be desired. This suggests that future studies will require a more rigorous and scientific approach. We suggest using multiple clinical factors that have been confirmed by clinical studies to be associated with COVID-19 severity as predictor variables, dividing the development data set and internal validation data sets according to the time of admission of patients with COVID-19, and using data from other hospitals to assess the performance of the model. However, there is no denying that DL can help clinicians quickly identify patients that are severely ill and detect potentially serious cases early, leading to earlier treatment and more efficient health care systems.