This umbrella review was performed in line with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [18] and Joanna Briggs Institute (JBI) guidelines (Multimedia Appendix 1) [19]. The predefined protocol was registered in the international open-access Prospective Register of Systematic Reviews (PROSPERO; CRD42022364278).

The present systematic literature search was completed using the following databases: PubMed, Embase, Web of Science, Cochrane, and IEEE. The search terms were as follows: (“artificial intelligence” OR “machine learning” OR “deep learning” OR “neural network”) AND (“carcinoma” OR “tumor” OR “cancer” OR “neoplas*” OR “maligna*”) AND (“meta-analysis” OR “systematic review”). The search was limited to articles written in English and published from inception to June 19, 2024. For the full search strategy, please see Multimedia Appendix 2.

All systematic reviews with or without a meta-analysis that examined the noninvasive imaging diagnostic performance of AI technologies for cancers were eligible for inclusion. We also included reviews that had a systematic strategy for the literature search, even if the authors did not specify the article type [20]. Studies were excluded if they (1) focused on AI-based approaches for predicting the outcomes of interventions or the prognosis of cancers; (2) did not show at least one of the following measures of classifier performance: accuracy, sensitivity, specificity, or area under the curve (AUC); (3) did not involve noninvasive images, a systematic review, or interested outcomes; (4) were primary studies, scoping reviews, literature reviews, rapid reviews, criterial reviews, and other types of reviews; and (5) were conference abstracts and posters, commentaries, preprints, proposals, and editorials.

Following the removal of duplicates, 2 investigators (TTG and HLX) independently screened the titles and abstracts to identify potentially relevant articles. The full texts of these articles were then assessed for eligibility based on the inclusion and exclusion criteria. Disagreement among the 2 reviewers was resolved by discussion with a third reviewer (XJS).

Two investigators (XJS and QB) independently extracted valuable data, using a predefined data extraction sheet. The following data were extracted: (1) characteristics of eligible reviews, including first author, year of publication, country, cancer site, registered protocol (yes or no), followed guidelines, research question, database searched, language restrictions, numbers of retrieved studies and included studies, number of reviewers (study selection, data extraction, and quality assessment), quality assessment tool, and meta-analysis (yes or no); (2) features of models in the included reviews, including imaging modalities, dataset size, AI approach, classification algorithms, and type of validation; and (3) classifier performance in multiple system cancers, including accuracy (n), sensitivity (n), specificity (n), AUC (n), and I². Points of divergence were resolved by discussion among 2 authors (HLX and TTG).

Currently, AMSTAR 2 is the most widely used tool for evaluating the quality of systematic review methodologies [21]. However, the revision was not intended to deal with the special requirements of diagnostic systematic reviews [21]. Therefore, the methodological quality of the included articles was rated using only the JBI Critical Appraisal Checklist for Systematic Reviews and Research Syntheses [19]. This process was done independently by 2 investigators (MMX and WY), with group discussion when necessary.

The Grading of Recommendations, Assessment, Development, and Evaluation (GRADE) is the established tool for assessing the overall certainty of evidence [21-23]. For this review, we examined the following GRADE domains: (1) risk of bias in the individual studies, (2) inconsistency, (3) indirectness, (4) imprecision, and (5) publication bias. Based on these criteria, the meta-analytical evidence was classified as high, moderate, low, or very low. High and moderate certainty in evidence means that it is very likely or probable that the true effect lies close to the estimated finding, and low or very low certainty means that we have little or very little confidence in the finding [24,25]. This assessment was conducted by 2 independent researchers (QC and QB), with discussion and agreement for any differences.

Given the significant heterogeneity between the reviews in terms of AI classifiers, imaging modalities, characteristics of participants, settings of conducted studies, etc, a narrative-only summary was considered. Study findings will be presented in graphs in the same way as the syntheses are reported in the narrative text to facilitate the comparison of findings from each included study [26].

The detailed process for this umbrella review is presented in a schematic flowchart in Figure 1. The initial database search retrieved 3690 candidate publications, of which 2691 were screened following the removal of 999 duplicates. Based on the title and abstract, 2350 publications were excluded. The remaining 341 publications underwent a full-text examination to confirm eligibility. This resulted in a final total of 158 studies that met the inclusion criteria (Figure 1). The list of the excluded records during the process of full-text checking is provided in Multimedia Appendix 3.

The detailed characteristics of the included studies are presented in Figures 2 and 3 and Multimedia Appendices 4-6. More than half of the included studies (133/158, 84.2%) were published in 2021-2024 (Figure 2A). The included studies focused on cancers in the following 8 systems (Figure 2B): central nervous system (CNS) (n=15), head and neck (n=18), respiratory system (n=16), digestive system (n=43), urinary system (n=18), female-related systems (n=23), skin (n=18), and others (n=8). Among them, 1 review reported on both nervous system and respiratory system tumors. The included studies were conducted in 31 different countries, with half of them conducted in Asia (n=86) (Figure 2C). The most commonly searched database was PubMed (113/158, 71.5%) (Figure 2D). The number of retrieved and included studies in the meta-analyses or systematic reviews ranged from 279 to 117,464 and 3 to 32, respectively.

Different proportions of meta-analyses or systematic reviews had registered protocols, followed the PRISMA guidelines, restricted searches to English-language studies, and conducted quality assessment (Figure 3). Several criteria were mentioned for quality assessment, such as Quality Assessment of Diagnostic Accuracy Studies-2 (QUADAS-2), radiomics quality score (RQS), and Checklist for Artificial Intelligence in Medical Imaging (CLAIM). At least two independent reviewers carried out the study selection process, data extraction, and quality assessment in 106, 82, and 66 meta-analyses or systematic reviews, respectively. Seventy-seven meta-analyses or systematic reviews synthesized the data using quantitative analysis.

The studies included in the meta-analyses or systematic reviews used various imaging data to match the model. Among them, US, CT, and MRI were the most frequently adopted imaging approaches in most diseases. Moreover, dermoscopy in skin cancer, endoscopy in gastrointestinal tumors, and mammography in breast cancer were employed. A variety of algorithms were used in the included studies, and the most common ones were convolutional neural network (CNN) and support vector machine (SVM). Algorithms, including DenseNet, EfficientNet, RetinaNet, ImageNet, and Inception-v3, were frequently employed. Forty-nine meta-analyses or systematic reviews included articles that used both internal and external validation.

The JBI critical appraisal results are summarized in Figure 4 and Multimedia Appendix 7. Overall, 80.4% (127/158) of the meta-analyses or systematic reviews were of high quality, and the rest were of medium quality. Out of 11 criteria, the minimum number of criteria met was 5 and the maximum was 11. Criteria 1-3, relating to clear questions, appropriate criteria, and appropriate search strategy, were met by the vast majority of included meta-analyses or systematic reviews.

The results of the GRADE assessment are presented in Multimedia Appendix 8 and Figure 5. Overall, 139 pieces of evidence met the conditions of the GRADE. The certainties of evidence were as follows: high, 5.0% (7/139); moderate, 37.4% (52/139); low, 46.8% (65/139); and very low, 10.8% (15/139).

Overall, 158 meta-analyses or systematic reviews contained information on the diagnostic performance of AI for multisite cancers: brain, oral, nasopharyngeal, thyroid, lung, gastrointestinal, esophageal, liver, pancreas, renal, bladder, prostate, breast, cervical, ovarian, skin, chondrosarcoma, bone, adrenal, and soft tissue tumors. The performance of the AI models in diagnosing these cancers is presented in Multimedia Appendix 9.

Ideally, the diagnostic approach should be explored in the setting of its intended use. In reality, numerous studies were not representative of the general population and community practice, with studies being performed in an experimental environment. For example, to date, all published data investigating radiomics in the field of pancreatic neuroendocrine tumors were mainly single-centered and retrospective studies [182]. The datasets largely included lesions from patients recruited in specialist clinical settings, and the applicability of these classifiers remains theoretical, making it more difficult to replicate the findings [183]. Meanwhile, the number of positive images and negative images in some included studies was significantly different, which was highlighted in a review on gastrointestinal cancer [92]. As suggested in recently published guidelines [88,184], a step forward needs to be achieved. Both current and future studies need to be constantly refined and externally validated in multicentric, large-sample, randomized clinical trials. In this regard, public access to large databases of clinical and radiological correlated data will be instrumental [182]. The creation of large public image datasets with images as representative as possible of the world’s people to avoid racial bias is a major task in this research field [171,185].

The variety of algorithms used in the assessed studies was high, and among them, CNN and SVM were the most widely used DL and machine learning methods in classification steps. SVM was typically used for limited data, while CNN was preferred for larger datasets. This distinction reflects the researcher’s expertise and preferences but is primarily driven by data constraints [186]. Even if high-performance models generate reasonable results, data from different training environments can contain various types of bias and noise and may not be reproduced again [159,187], and technique challenges remain to be settled [170]. The ability of clinicians to accept and trust the outputs of an algorithm, when the decision-making process is not apparent or comprehensible to them, may prove to be an obstacle to adoption [188]. A vivid metaphor is that the algorithm is mostly like a “black box” [189]. Acceptability to medical professionals and regulatory agencies may be increased if there is enhanced understanding as to how an algorithm arrives at its decision. Moving forward, further research into so called “explainable AI” [190] may provide the necessary transparency, trust, and accountability desired by the health care profession.

It should be discussed that a wide range of metrics was employed to report diagnostic performance in AI studies. We noticed that the evaluations were not performed based on a single procedure, and many studies used their own indicators. For example, some studies reported only 1 indicator; hence, it is not possible to know whether the study performed better in terms of other indicators [186]. There is no uniform standard and evaluation standard for the specific AI model design. In view of certain parameters being comprehensive and useful in clinical practice, it is an ideal research process to compare the accuracy, sensitivity, specificity, or AUC of some models with those of other methods and datasets in studies [100].

In the domain of tumor imaging diagnosis with the application of AI, several notable trends have emerged. Regarding AI technology trends, the continuous refinement and enhancement of DL algorithms, especially the advancement of CNN, have significantly improved the precision in analyzing features from tumor images. For instance, advanced architectures, such as ResNet and DenseNet, have achieved remarkable success in enhancing diagnostic accuracy [37,51,52]. Moreover, the use of transfer learning and pretrained models has become increasingly prevalent. By leveraging pretrained models on large-scale general image datasets like ImageNet and fine-tuning them for tumor imaging data, training time and data requirements have been considerably reduced [43]. In terms of target task categories, tumor detection has emerged as a critical objective, where AI techniques strive to identify early tumor indications from subtle image variations [87,114]. The classification and grading of tumors have also received significant attention, helping physicians in formulating more individualized treatment plans [191]. Additionally, in the process of literature screening, we found that comparing pre- and posttreatment imaging to determine whether the tumor has shrunk or remained stable, has become an important area of focus [192,193].

The emergence of generative AI and transformer-based technology has brought new possibilities. Generative AI can generate realistic tumor simulation images, providing a novel approach for expanding training data and aiding physicians in better comprehending tumor morphological changes [194]. Transformer-based technology excels in handling sequential data, such as time series in medical imaging, contributing to capturing the dynamic development of tumors [195,196]. When it comes to foundation models, the exploration of large-scale ones, such as large language models, in the medical field is on the rise. Integrating image and related clinical text information enables more comprehensive diagnostic support [197-199]. Multi-modal fusion foundation models are emerging, integrating information from multiple imaging modalities like CT, MRI, and PET to enhance diagnostic accuracy and reliability [200]. As mentioned above, the trends in AI for tumor imaging diagnosis are moving toward greater precision, comprehensiveness, personalization, and efficiency, offering new prospects for the detection and further precise treatment of tumors.

According to the findings of this study, the AI systems in cancer detection follow 2 directions: (1) as concurrent assistants aiding clinicians in diagnosis and (2) as standalone systems providing an independent assessment. In the former case, the AI system is used as an aid during interpretation [201]. In particular, when using images for diagnosis, factors, such as physician fatigue, stress, and limited experience, may lead to neglected or misdiagnosed diseases. In contrast, AI can continuously provide reliable performance over a short period of time, has the potential to compensate for limited human capabilities, can prevent mistakes made by doctors in clinical practice, and can facilitate the training and education of less experienced endoscopists [116]. In the latter case, AI can play an important role in areas where the economy is underdeveloped and medical resources are relatively scarce. However, a major concern is the reproducibility, reliability, and usability of the diagnostic capability of AI, as false-negative results will lead to misdiagnosis and false-positive ones may result in overdiagnosis [159,202]. In this respect, AI may not completely replace doctors, and human beings and machines working together in harmony is an ideal state that can result in optimum performance.

As accuracy becomes more important, consolidating the information found in various forms of clinical data is vital in future clinical practice [96]. If purely based on image modeling, it is evident that the approach does not take into account all the information a clinician would rely on to evaluate a difficult examination [203]. The addition of clinical data, such as race, age, and gender, as inputs for classifiers may help to increase classification accuracy. This supplemental data could benefit clinicians in their decision-making [171]. Furthermore, AI has shown its advantages in computational power and learning capacity. An AI model is adept at integrating a lot of information from most data, which has the potential to reduce the workload of clinicians substantially, as it is difficult for them to integrate complex data manually [96]. These aspects should be included in future work.

Processing and modeling medical images have traditionally represented complex tasks requiring multidisciplinary collaboration. Technically, the lack of a suitable information technology infrastructure in medical practices, the absence of high-quality curated data, and difficulties to access and exchange data are inhibiting the translation of integrated diagnostics into clinical routine and even research. Looking at the issue from a legal perspective, the sensitive issues of data privacy and security, patient consent, and autonomy must be fully considered. This means that from a legal perspective, data acquisition, storage, transfer, processing, and analysis will have to comply with all laws, regulations, and further legal requirements. In addition, the law and its interpretation and implementation must constantly adapt to the evolving state-of-the-art in technology [204,205]. We have observed a growing trend of clinical studies using commercial software for image processing [12]. The cost-effectiveness evaluation of AI in diagnosis is also warranted. Multidirectional interactions should be implemented among physicians, developers, legislators, and economists to foster multidisciplinary collaboration to tackle these challenges jointly [71]. When malpractice cases involving medical AI applications arise, multidisciplinary teams need to provide necessary suggestions [190].

Multiple tools were employed in our included systematic reviews to comprehensively assess the quality of the included articles. The studies were evaluated by the RQS [206], CLAIM [207], Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statement [208], Image Biomarker Standardization Initiative (IBSI) guideline [209], and QUADAS-2/AI tool [116,210]. It should be noted that each tool has its own focus. For example, the RQS does not take into account sample size or how the model actually performs but instead represents an evaluation of both how rigorous model development is and how impactful the study may be to the field [211]. The IBSI was able to produce and validate reference values for radiomics features. The TRIPOD is a similar example that aims to promote the transparent reporting of diagnostic accuracy model studies. There are suggestions from researchers that CLAIM may guide the update of the TRIPOD and RQS, because it not only includes general reporting criteria but also allows extra distinction of unique shortness in DL. Considering the overlapping items and high correlation between these tools, researchers can choose an appropriate one or a combination as the case may be. We are also confident that relevant organizations or scholars will continue to refine existing tools to better guide and evaluate relevant research.

The strengths of this umbrella review include an exhaustive search of medical and engineering databases; assessment of methodological quality using the JBI tool; independent screening, quality assessment, and data extraction processes; and methodological rigor in the conduct of the review achieved by following the PRISMA guidelines. We also identified challenges that will need to be overcome for the technology to be implemented into daily clinical practice [94]. Umbrella reviews represent the highest level of evidence synthesis currently available and are becoming increasingly influential in biomedical literature [212]. Evidence from the umbrella review highlights the massive opportunity for AI image–based cancer detection. Remarkably, by identifying potential biases and methodological limitations, the information provided can raise the awareness of health care decision makers regarding the strengths and weaknesses of available AI diagnostic tools.

Several limitations of this review merit consideration. First, this review included a synthesis of evidence from existing systematic reviews or meta-analyses [116]. Therefore, cancers that were not systematically reviewed in the pre-existing literature were not included in this umbrella review. The lack of generalizability and the problem of out-of-distribution lesions, such as rare or more harmful disease categories, are limitations that are not addressed by current studies. Second, we could not examine the algorithms that were built without publication, such as those developed for private companies or for business usage [12]. Third, this umbrella review was limited by substantial variability in terms of the characteristics of the patients included, method quality, and study design, which precluded a full statistical analysis. Furthermore, the GRADE assessment revealed that the overall quality of nearly half of the evidence was low (47%). Since secondary research is highly dependent on the quality and availability of primary studies, both this umbrella review and previous systematic reviews were considerably restricted in improving the current understanding of cancer diagnosis.

Conclusively, the main points discussed and investigated in this review can be summarized as follows. AI algorithms have shown remarkable efficacy in the noninvasive imaging diagnosis of tumors. Notwithstanding the progress, certain challenges persist. The data used in current AI models often have limitations in terms of quantity, quality, and diversity, which can compromise the model’s generalization ability. Additionally, the lack of transparency and interpretability of some AI systems poses difficulties for clinicians in understanding and trusting the diagnostic results. In the future, efforts should be directed toward enhancing data quality and diversity, developing more interpretable AI models, and conducting large-scale, multicenter clinical trials to validate the practical effectiveness of these technologies. While AI holds tremendous promise in noninvasive tumor imaging diagnosis, a concerted effort from the research community, clinicians, and policymakers is required to overcome the existing hurdles and translate this potential into improved patient outcomes and health care delivery.