This study was conducted in accordance with the statement of the PRISMA (Preferred Reporting Items for a Systematic Review and Meta-analysis) of DTA Studies [9]. The study protocol was registered at the International Prospective Register of Systematic Reviews (PROSPERO) database before initiation of the systematic review (#CRD42021253454). Approval from the institutional review board of the Chuncheon Sacred Heart Hospital was waived.

The authors established searching formulae using keywords related to the performance of CAD models in the detection of ulcer or hemorrhage using WCE images. Medical Subject Headings (MeSH) terminology keywords were used for the establishment of searching formulae (Textbox 1).

Two authors, CSB and JJL, independently performed a core database search of MEDLINE through PubMed, Web of Science, and Cochrane Library using pre-established search formulae from inception to May 2021. Duplicate articles were excluded. The titles and abstracts of all identified articles were reviewed, and irrelevant articles were excluded. Full-text reviews were subsequently performed to determine whether the pre-established inclusion criteria were satisfied in the identified literature. The references of relevant articles were also reviewed to identify any additional studies. Any disagreements of results in the searching process between CSB and JJL were resolved by discussion or consultation with the other author (GHB).

The literature included in this systematic review met the following inclusion criteria: designed to evaluate the diagnostic performance of CAD models for gastrointestinal ulcers or hemorrhage based on WCE images; presentation of the diagnostic performance of CAD models, including sensitivity, specificity, likelihood ratios, predictive values, or accuracy, which enabled the estimation of TP, FP, FN, and TN values of CAD models; and written in English. The exclusion criteria were as follows: narrative review articles, studies with incomplete data, systematic review or meta-analyses, comments, proceedings with only an abstract, or study protocols. A full publication with PDF files of available proceedings was considered to be a full article. Articles meeting at least 1 of the exclusion criteria were excluded from this study.

The methodological quality of the included articles was assessed by CSB and JJL using the second version of Quality Assessment of Diagnostic Accuracy Studies (QUADAS-2). This tool comprises 4 domains, including “patient selection,” “index test,” “reference standard,” and “flow and timing,” with the first 3 domains possessing an “applicability” assessment. CSB and JJL assessed each part as having either a high, low, or unclear risk of bias [10].

CSB and JJL independently extracted the data from each included article and cross-checked the extracted data. If data were unclear, the corresponding author of the study was contacted by email to obtain insight into the original data set. A descriptive synthesis was made by a systematic review process, and DTA meta-analysis was conducted if the included studies were sufficiently homogenous.

The primary outcomes were the TP, FP, FN, and TN values in each study. For the CAD of gastrointestinal ulcers or hemorrhage using WCE images, the primary outcomes were defined as follows: TP, the number of patients with a positive finding by a CAD model and who had ulcers or hemorrhage as evidenced by WCE images; FP, the number of patients with a positive finding by a CAD model and who did not have ulcers or hemorrhage based on WCE images; FN, the number of patients with a negative finding by a CAD model and who had ulcers or hemorrhage as evidenced by WCE images; and TN, number of patients with a negative finding by a CAD model and who did not have ulcers or hemorrhage based on the WCE images. With these definitions, TP, FP, FN, and TN values were calculated for each included study.

For additional analysis, such as subgroup analysis or meta-regression, the authors extracted the following variables from each included study: published year, geographic origin of the data (ie, Western vs Asian vs public data or unknown), type of CAD models, type of endoscopic images, number of total images included, type of test data sets (internal test vs external test), and the target disease (ulcer vs erosions, hemorrhage, or angioectasias).

The hierarchical summary receiver operating characteristic (HSROC) method was used for the DTA meta-analysis [11]. A forest plot of the sensitivity and specificity and a SROC curve were generated. The level of heterogeneity across the included articles was determined by the correlation coefficient between logit-transformed sensitivity and specificity by the bivariate method [12]; for this, the asymmetry parameter was β, where β=0 corresponds to a symmetric ROC curve in which the diagnostic odds ratio (DOR) does not vary along the curve according to the HSROC method [11]. A positive correlation coefficient and a β value with a significant probability (P<.05) indicate heterogeneity between the studies [11,13]. Visual examination of the SROC curve was also performed to identify heterogeneity. Subgroup analysis by univariate meta-regression using the modifiers identified during the systematic review was also performed to identify the reasons for heterogeneity. STATA software version 15.1 (StataCorp), including the packages “metandi” and “midas,” was used for the DTA meta-analysis. Publication bias was evaluated using the Deeks funnel plot asymmetry test. For the subgroup analyses including less than 4 studies, the Moses-Shapiro-Littenberg method [14] was adopted using Meta-DiSc 1.4 (XI Cochrane Colloquium) because the “metandi” and “midas” packages require the inclusion of a minimum of 4 studies for DTA meta-analysis.

A total of 254 studies (80 studies for the CAD of gastrointestinal ulcers or erosions and 174 studies for the CAD of gastrointestinal hemorrhage using WCE) were identified following a literature search of 3 databases. Fifteen studies were additionally identified by manual screening of bibliographies. After excluding duplicate studies, additional articles were excluded after a review of titles and abstracts. Full-text versions of the remaining 54 and 118 articles were obtained and thoroughly reviewed based on the aforementioned inclusion and exclusion criteria in each topic. Among these, 133 articles were excluded from the final enrollment. Finally, 20 studies [15-34] for the CAD of gastrointestinal ulcers or erosions and 19 [17,19,24,34-49] studies for the diagnosis of gastrointestinal hemorrhage were included in the systematic review. A flowchart of the selection process is shown in Figure 1 and 2.

Among the 20 studies [15-34] for the CAD of gastrointestinal ulcers or erosions using WCE, a total of 40,809 images were identified (14,866 cases vs 25,943 controls) for the assessment of the diagnostic performance. Given that the duplicate data were identified (Karargyris et al in 2009 [15] and Karargyris et al in 2011 [20]), all the analyses used the data of 19 studies [15-19,21-34] (a study by Karargyris et al in 2011 [20] was omitted in the meta-analysis).

Ten studies [16-18,21,25-27,29,31,32] used endoscopic images from Asian populations, and 3 studies [22,23,33] used endoscopic images from Western populations. However, 6 studies [15,19,24,28,30,34] used public database images or an unknown source of images. Regarding the type of CAD model, a deep neural network or convolutional neural network was used in 9 studies [16-18,27-29,31-33], and machine learning–based models were used in 10 studies [15,19,21-26,30,34]. Most of the included studies [15-28,30-34] presented the diagnostic performance for the intestinal ulcers. However, the study by Aoki et al [29] presented an indistinguishable performance for the intestinal ulcers or erosions, and the study by Fan et al [27] presented a separate performance for the intestinal ulcers and erosions. Therefore, subgroup analyses were performed for the target lesions. Detailed clinical features of the included studies are presented in Multimedia Appendix 1.

Among the 19 studies [17,19,24,34-49] for the diagnosis of gastrointestinal hemorrhage using WCE, a total of 41,323 images were identified (6952 cases vs 34,371 controls) for the assessment of the diagnostic performance.

Five studies [17], [35], [44], [48], [49] used endoscopic images from Asian populations, and 1 study [47] used endoscopic images from Western populations. However, the remaining 13 studies [19,24,34,36-43,45,46] used public database images or an unknown source of images. Regarding the type of CAD model, the deep neural network or convolutional neural network was used in 8 studies [17,35,39,43,46-49], and machine learning–based models were used in 11 studies [19,24,34,36-38,40-42,44,45]. Most of the included studies [17,19,24,34-46,48] presented the diagnostic performance for intestinal hemorrhage. However, studies by Leenhardt et al [47] and Tsuboi et al [49] presented the performance for angiodysplasias. Therefore, subgroup analyses were performed for the target lesions. Detailed clinical features of the included studies are presented in Multimedia Appendix 2.

The quality and quantity of baseline training data are important because the CAD models are established using learning features of the baseline training data. A sufficient number of training images are required for the establishment of practical CAD models, and endoscopic specialists should participate in the labeling work for the accurate preparation of training data. We also could not guarantee the quality of images in public databases searched on the internet. We determined that proper learning requires at least 30 training images (quantity standard) from real clinical hospital data (quality standard) labeled by an endoscopic specialist (quality standard). If both quality and quantity standards were satisfied, there was considered to be a low risk of bias in the patient selection domain. If only 1 of these quality or quantity standards was satisfied, there was considered to be an unclear risk of bias. If both were not satisfied, there was considered to be a high risk of bias.

In terms of the CAD of gastrointestinal ulcers or erosions in WCE, only 7 studies [25-27,29,31-34] were rated as low risk of bias, 9 studies [16-18,21-24,30,34] were rated as unclear risk of bias, and 3 studies [15,19,28] were rated as high risk of bias in the “patient selection” domain. The remaining domains were rated as having a low risk of bias in all the included studies (Figure 3). Therefore, classification of methodological quality in the “patient selection” domain was adopted as a modifier in the subgroup or meta-regression analysis.

In terms of the CAD of gastrointestinal hemorrhage in WCE, only 3 studies [44,47,49] were rated as having a low risk of bias, 10 studies [17,24,34,35,37,38,40,45,46,48] were rated as having an unclear risk of bias, and 6 studies [19,36,39,41-43] were rated as having a high risk of bias in the “patient selection” domain. The remaining domains were rated as having a low risk of bias in all the included studies (Figure 4). Therefore, the classification of methodological quality in the “patient selection” domain was adopted as a modifier in the subgroup or meta-regression analysis.

Among the 20 studies [15-34] for the meta-analysis of the CAD of gastrointestinal ulcers or erosions using WCE, the area under the curve (AUC), sensitivity, specificity, positive likelihood ratio, negative likelihood ratio, and DOR were 0.97 (95% CI 0.95-0.98), 0.93 (95% CI 0.89-0.95), 0.92 (95% CI 0.89-0.94), 11.2 (95% CI 8.6-14.7), 0.08 (95% CI 0.05-0.12), and 138 (95% CI 79-243), respectively (Multimedia Appendix 3 and Figure 5). The SROC curve is illustrated in Figure 6. To investigate the clinical utility of the CAD models, Fagan’s nomogram [50] was generated. Positive findings indicated that gastrointestinal ulcers or erosions were detected by the CAD models, while negative findings indicated that gastrointestinal ulcers or erosions were not detected by the CAD models. Assuming a 23% prevalence of gastrointestinal ulcers or erosions, Fagan’s nomogram showed that the posterior probability of ulcers or erosions was 76% if the finding of the CAD model was positive and that the posterior probability of ulcers or erosions was only 3% if the finding of the CAD model was negative (Figure 7).

Among the 19 studies [17,19,24,34-49] for the meta-analysis of the CAD of gastrointestinal hemorrhage in WCE, the AUC, sensitivity, specificity, positive likelihood ratio, negative likelihood ratio, and DOR were 0.99 (95% CI 0.98-0.99), 0.96 (95% CI 0.94-0.97), 0.97 (95% CI 0.95-0.99), 38.3 (95% CI 19.6-74.8), 0.04 (95% CI 0.03-0.07), and 888 (95% CI 343-2303), respectively (Multimedia Appendix 4 and Figure 8). The SROC curve is illustrated in Figure 9. Positive findings of Fagan’s nomogram indicated that gastrointestinal hemorrhage was detected by the CAD models. Negative findings indicated that gastrointestinal hemorrhage was not detected by the CAD models. Assuming a 10% prevalence of small intestinal hemorrhage in all gastrointestinal bleeding [51], Fagan’s nomogram showed that the posterior probability of small intestinal hemorrhage was 81% if the finding of the CAD model was positive and that the posterior probability of small intestinal hemorrhage was only 0.5% if the finding of the CAD model was negative (Figure 10).

For the CAD of gastrointestinal ulcers or erosions in WCE, we first observed a positive correlation coefficient between the logit-transformed sensitivity and specificity (r=0.28) in the bivariate model analysis. However, an asymmetric β parameter in the HSROC model showed an nonsignificant P value (P=.15), implying that heterogeneity was not present among the studies. Second, a coupled forest plot of sensitivity and specificity was observed (Figure 5). Compared with the enrolled studies, the study by Karargyris et al (2009) [15] showed lower sensitivity and specificity. This study was found to have a high risk of bias in the methodology quality assessment (Figure 3). Therefore, subgroup analysis was carried out according to the methodological quality, and the performance was robust although slightly higher values were observed in the studies of high methodological quality (Multimedia Appendix 3). Third, the shape of the SROC curve for the gastrointestinal ulcers or erosions in WCE was symmetric, and the 95% prediction region was not wide (Figure 6). Fourth, meta-regression using modifiers identified in the systematic review was conducted, and published year, number of training images, and target disease (ulcer vs erosion) were found to be the source of heterogeneity (published year: P=.04; number of training images: P=.02; target disease ulcer vs erosion: P=.38; type of endoscopic image: P=.01). Finally, a subgroup analysis based on the potential modifiers was performed, and the overall performance of the studies published within 10 years (vs studies published more than 10 years ago) and studies with more than 100 training images (vs studies with fewer than 100 training images) showed higher values (Multimedia Appendix 3).

For the CAD of gastrointestinal hemorrhage in WCE, we first observed a positive correlation coefficient between the logit-transformed sensitivity and specificity (r=0.48) in the bivariate model analysis. However, an asymmetric β parameter in the HSROC model showed an nonsignificant P value (P=.06), implying that heterogeneity was not present among the studies. Second, a coupled forest plot of sensitivity and specificity was observed (Figure 8), and there was no significant outlier. Third, the shape of the SROC curve for the gastrointestinal ulcers and erosions in WCE was symmetric, and the 95% prediction region was not wide (Figure 9). Fourth, a meta-regression using the modifiers identified in the systematic review was conducted, and published year, number of training images, and target disease (hemorrhage vs angioectasia) were found to be the source of heterogeneity (published year: P<.01; number of training images: P=.04; target disease hemorrhage vs angioectasia: P<.01). Finally, a subgroup analysis based on the potential modifiers was performed, and the overall performance of the studies published within 10 years (vs studies published more than 10 years ago) and studies with more than 100 training images (vs studies with fewer than 100 training images) showed higher values (Multimedia Appendix 4).

The Deeks funnel plot of studies for the gastrointestinal ulcers or erosions in WCE exhibited a symmetrical shape with respect to the regression line (Figure 11), and the asymmetry test showed no evidence of publication bias (P=.77). The Deeks funnel plot of studies for the gastrointestinal hemorrhage in WCE exhibited a symmetrical shape with respect to the regression line (Figure 12), and the asymmetry test showed no evidence of publication bias (P=.93).

In this study, CAD models showed high performance values for the diagnosis of gastrointestinal ulcers or erosions and hemorrhage in WCE images. Practical values in Fagan’s nomogram indicated the potential to use CAD models in clinical practice. Although the main analyses found some heterogeneity among the included studies, the meta-regression showed the common reasons for heterogeneity (published year, number of training images, and target disease—ulcers vs erosions and hemorrhage vs angioectasia), and subgroup analyses demonstrated that recently published studies (vs studies published more than 10 years ago) with a greater amount of training data (vs studies with fewer than 100 training images) showed better performance of CAD models. Thorough subgroup analyses indicated the robust quality of the evidence.

Interpretation of WCE images is an important task for gastroenterologists. Due to the fact that WCE presents the images of the whole gastrointestinal tract, lesions that are difficult to detect with conventional endoscopy can be identified. Diminutive but important culprit lesions also can be found in the WCE examination. The noninvasive nature of this examination and patients’ comfort have also promoted the use of this technique in the diagnosis of obscure gastrointestinal hemorrhage or small intestinal disorders. However, the interpretation process is tedious. At least 30 to 120 minutes of reading time is required for the endoscopists [1-3]. It is necessary to maintain concentration throughout the reading time so as not to miss important lesions. CAD models have the potential to automate the reading process of WCE with their high diagnostic performance, especially for sensitivity and specificity. The overall performance is slightly higher for gastrointestinal hemorrhage than for ulcers or erosions. It is presumed that this is because red-colored blood is easier to distinguish than are white- or yellow-colored ulcers or erosions, which are similar to the color of the background mucosa for the pixel-based or red-green-blue spectrum–based feature learning of CAD.

In the context of the learning way of CADs, neural network–based CAD models showed a slightly higher performance than that of traditional machine learning–based CAD models (Multimedia Appendices 3 and 4). CNN is not always better than machine learning for accurate classification. However, image recognition with local feature extraction can be highly optimized with its complex layers and deep nodes calculations and dimensional reductions for neural network CAD models. Considering that the machine learning–based models in the included studies used color or textures features in the images of WCE, neural network–based models might focus on the other local features or combined features, such as the shape of the lesions or feature differences between the lesions and background mucosa. Explainable artificial intelligence analyses are on the rise, and the application of this technique would provide a method of determination in the CAD models [52].

Although meta-analyses of same topic have already been published, this study was conducted to evaluate the DTA of CAD models for gastrointestinal ulcers or hemorrhage using WCE images with a standard methodology (Table 1) [7,8]. Although previous studies also reported the high performance of CAD models, many important articles were omitted, the heterogeneity between studies was determined by I² statistic (which is used in interventional meta-analysis), methodological quality assessments were omitted, and the publication bias was also not assessed.

Despite the robust evidence in this meta-analysis, several inevitable limitations were identified. First, all the performance data were only measured in an internal-test setting in each included study. Modeling is an assumption that observations follow certain statistical rules, and external validation is a method to check whether this assumption is correct or generalizable. Therefore, the confirmation of performance in the established CAD models with unused data in the training or internal testing process is essential [53]. However, no single study conducted performance verification in an external validation setting. Second, the definition of intestinal ulcers or erosions was vague. Erosion usually refers to damage that is limited to the mucosa (loss of the epithelium but with the basement membrane or lamina propria being intact). However, the definition of ulcers usually involves more extensive loss of the mucosa beyond the lamina propria. Although the discrimination between these 2 conditions is not perfect under visual inspection, there was no clear definition in the included studies. This can lead to the underestimation or overestimation of the performance of CAD models. Third, many studies used baseline training data from a public database, and we could not guarantee the quality of images in the public databases available from the internet. The diagnostic performance of the CAD models can only be valid for the population under evaluation and depends on the prevalence of target conditions for the selected population (so-called spectrum bias or class imbalance) [2,54]. This class imbalance was not considered in the included studies. Most of the studies except for 1 [34] applied a 1:1 to 1:4 ratio (target condition:normal mucosa) of the training data set. However, Kundu et al [34] used 31 ulcer images and 1617 normal mucosal images (about a 1:52 ratio) and 65 bleeding images and 1617 normal mucosal images (about a 1:25 ratio) in the training data set. Considering that the method of establishing artificial intelligence models is changing from a model-centric (ie, change or optimize the model to improve performance) to a data-centric approach (ie, systematically change the distribution of the quality of data to improve performance), model establishment that takes into account spectrum bias is required. Overall, qualified training data with clear definitions and a focus on external validation-oriented performance CAD model establishment are required and expected for future perspectives in this topic.

In conclusion, CAD models showed high performance for the optical diagnosis of gastrointestinal ulcers and hemorrhage in WCE.