This section defines the terminology and classification system for xAI methods reviewed in cancer imaging studies. The objectives are (1) to provide a standard way of referring to “interpretability,” “explainability,” and “transparency” in the context of this review; (2) to group xAI techniques commonly reported in studies into reproducible families of methods; and (3) to identify how each study documented its intended application of explainability and whether any evaluations of that explainability were conducted.

The concepts and taxonomic distinctions referenced in this section are based on previous research related to the field (eg, Arrieta et al [34], Holzinger et al [35], and Roscher et al [36]). In other words, our goal in mapping each included study to this established landscape is to use the existing framework for reference.

This review analyzed 371 studies (Multimedia Appendix 3) that used various terms to describe “interpretability, “explainability,” and “transparency” for presenting their research findings with a wide range of xAI techniques. For each study included, we mapped its explainability approach onto 4 dimensions. First, we recorded the xAI strategy, distinguishing whether the approach was inherently interpretable, post hoc, or a hybrid of the two. Second, for studies that used post hoc xAI, we assigned one or more post hoc method categories, drawing on the families defined by Arrieta et al [34], which include text explanations, visual explanations, local explanations, explanations by example, explanations by simplification, and feature relevance explanations. Third, we documented the model dependence of each technique, classifying methods as either model-specific or model-agnostic. Finally, we captured the scope of the explanation, noting whether it was intended to be local (instance-level), global (model- or dataset-level), or a combination of both.

Explainability and interpretability are 2 terms that can be found used interchangeably in some cases, although some literature suggests they should be distinguished, as they convey different meanings [37,38]. Both support transparency, which refers to clarity regarding the model’s structure, data, and assumptions, a cornerstone of trustworthy AI [39]. We use the following working definitions to prevent ambiguity throughout this review.

Interpretability refers to an intrinsic (passive) property of a model: the extent to which a human can directly understand how inputs are mapped to outputs without requiring an additional explanatory mechanism. Roscher et al [36] describe interpretability as the inherent comprehensibility of a model. Linear regressions or shallow decision trees (DTs) are interpretable by design because their parameters map directly to explicit features, and Arrieta et al [34] relate this notion to transparency.

We use explainability to refer to an active property: the ability of a system to generate reasons, evidence, or visual and linguistic justifications for its outputs in a form intended to be understandable to a target audience. Explainability may be inherent to the architecture (eg, a model that surfaces explicit prototypes or attention maps as part of its forward pass), or it may be added after training via post hoc analysis of an otherwise opaque model.

Transparency involves openness in model structures, data, and design assumptions. Transparency—informally the opposite of opacity—varies by context and user and can be analyzed across 5 dimensions—human involvement, data, the model, inferencing, and algorithmic presence [40]. Also, according to Arrieta et al [34], which is the one that guides this review, can be divided into 3 levels:

These distinctions matter because the terms “interpretable,” “explainable,” and “transparent” are not used consistently. In several instances across the literature, authors describe their approach as “transparent” primarily because the underlying model class is simple (eg, a shallow DT or sparse linear model). Other studies provide only post hoc saliency maps for a deep neural network and still refer to the solution as “transparent.”

In our review, we do not assign a transparency rating to each research study to measure the levels of simulatability, decomposability, or algorithmic transparency. Instead, we record (1) which model architecture was used and (2) whether interpretability is claimed to be inherent or is provided through a separate post hoc explanation method. This allows us to report, in the Results section, how often explainability is treated as a built-in design property versus an add-on.

To compare explainability strategies across studies, we group reported methods along these axes widely discussed in xAI research and directly aligned with our extraction fields: xAI method, type of xAI, and aim of interpretability or explainability.

A central distinction in this taxonomy is whether a study uses an interpretable-by-design model, a post hoc explainability technique, or a combination of both. Interpretable-by-design approaches construct the model so that its internal decision process is directly understandable. Examples include linear models built over predefined features, shallow DTs, rule-based systems, and prototype-based architectures in which predictions are explicitly linked to reference exemplars; in these cases, the explanation is embedded within the model itself. Post hoc explainability methods, by contrast, are applied after training to elucidate the behavior of otherwise opaque models, typically through outputs such as saliency maps, pixel or feature importance scores, or simplified local surrogate models. Hybrid approaches appear in studies that use an inherently interpretable model while also incorporating an additional post hoc xAI method.

During extraction, we map each study to one or more of these categories, based on how they describe their approach in ML or DL method used, type of xAI, and xAI method fields.

We then classified explainability techniques based on whether they depend on the internal structure of a particular model class. Model-specific methods rely on architecture-specific information, such as gradients, activation maps, attention scores, or layer relevance. Examples include gradient-weighted class activation mapping (Grad-CAM) and its variants for convolutional neural networks, attention-weight visualization in transformer architectures, and layer-wise relevance propagation. Because they incorporate internal model characteristics, these approaches generally cannot be applied unchanged to other model families. In contrast, model-agnostic methods treat the model as a black box and infer explanations by probing input-output behavior. Representative techniques include local interpretable model-agnostic explanations (LIME) [41], Shapley Additive Explanations (SHAP) [42], perturbation-based feature importance, occlusion sensitivity analyses, and locally fitted surrogate models for individual predictions. Since they do not depend on internal model layers, these methods can, in principle, be applied across a wide range of classifiers or regressors, including ensemble architectures.

We have separated our methods into local explanations (explanations for a single prediction, such as heatmaps of a suspected malignancy region in 1 patient) and global explanations (overview of model performance across the entire dataset, such as top-ranked lists of the most influential features). For each method, we have documented how it describes its own scope as local, global, or both. We note that “local explanation” appears in xAI categorization by Arrieta et al [34], where “local” refers to a family of methods focused on sample-by-sample justification, typically using surrogate models of local data.

Finally, for studies that use post hoc explainability, we assign each reported method to one or more method families, which follow the taxonomy presented in the study by Arrieta et al [34].

Individual studies may report more than one category (eg, both saliency maps and a local surrogate model). We record all categories. In the Results section, we report the prevalence of each category across cancer types and imaging modalities.

xAI methods have been applied to a wide range of cancer types, demonstrating their versatility in oncologic imaging. Breast cancer was the most extensively studied (n=112/371, 30.2%), followed by lung cancer (n=87/371, 23.5%) and brain or CNS tumors (n=56/371, 15.1%). Together, these three categories accounted for nearly 70% (255/371) of all reviewed studies, consistent with their global incidence and mortality burden and with previous bibliometric analyses of AI in oncology [1,43].

Moderately represented cancer types included prostate (22/371, 5.9%), thyroid (19/371, 5.1%), head and neck (16/371, 4.3%), and liver (hepatocellular and related; 14/371, 3.8%). Smaller groups addressed pancreatic (7/371, 1.9%), colorectal (5/371, 1.3%), renal or kidney (5/371, 1.3%), and cervical (4/371, 1.1%) malignancies. Other sites, such as stomach, bone, ovary, esophagus, hematologic, skin, endometrial, and bladder cancers, were examined in 5 or fewer studies (<1% each).

Within the “Other” category (n=24), 6 studies (6/371, 1.6%) addressed multidisease or nonorgan-specific contexts, including salivary gland tumors, metastasis or lymph node prediction, gastrointestinal stromal tumors, and mixed cancer cohorts that combined, for example, breast, brain, and cardiovascular imaging. Collectively, these studies highlight ongoing efforts to develop generalizable and cross-site explainable AI frameworks that extend beyond single-organ applications.

Across all included studies, CT (n=139, 37.5%) and MRI (n=104, 28%) were the most common primary modalities, followed by ultrasound (n=63, 17%) and mammography (n=46, 12.4%). Multimodal designs were present but were assigned to a primary modality (or “Other or multiple,” n=3, 0.8%) for counting in Table 1. X-ray and PET-CT were less common (n=8 each, 2.2%) but remain relevant for specific clinical contexts, particularly thoracic and metastatic imaging (Figure 2).

Over time, CT remained the most common modality, particularly in 2023 (56/171, 32.7%) and 2024 (83/200, 41.5%; Figure 2). MRI remained the second most common modality across the period. Ultrasound and mammography contributed a substantial minority of studies, particularly in later years, reflecting continued interest in breast-focused imaging and multimodality-based workflows. PET-CT and x-ray appeared less frequently but remained present in clinically relevant contexts.

Because some studies used multiple models, we assigned a single main modeling approach to each study. Categories are mutually exclusive: ML, DL, hybrid, or “atypical.” We labeled a study as hybrid when prediction relied on both DL and classical ML (eg, a Convolutional Neural Network [CNN] feature extractor with an extreme gradient boosting [XGBoost] for inference) or when the architecture explicitly integrated multiple families; otherwise, we labeled by the dominant predictive component (eg, end-to-end CNN=DL, radiomics plus gradient boosting=classical ML). Counts, therefore, sum to n=371 and refer to studies, not model instances.

In the majority of studied articles (n=260/371, 70.1%), DL architectures are used (Figure S2 in Multimedia Appendix 4 and Table 4). In most cases, they serve as the main analytical components of the analyzed studies; in other cases, they are part of hybrid pipelines where classical ML algorithms are used for secondary classification or feature aggregation. Within DL, CNN-based families (eg, Residual Network, Visual Geometry Group, and Densely Connected Convolutional Network) remain dominant for image-level tasks, U-Net variants are the default for segmentation, and transformer or attention architectures rose notably in 2023, aligning with self-attention maps as native interpretability; CNNs still constitute the largest DL subgroup (149/260, 57.3%). In hybrid CNN-to-ML setups, the DL block extracts spatial or contextual features while the ML block (eg, support vector machine [SVM], random forest [RF], and XGBoost) provides interpretable decision rules or feature rankings.

Classical ML algorithms (67/371, 18.1%) are still used frequently, mostly in analyses focused on radiomics or as comparative baselines. Models commonly selected for interpretability include RFs and SVMs (feature importance and margin or kernel effects), while since 2023 gradient-boosting families (Categorical Boosting, Light Gradient Boosting Machine, and XGBoost) are frequently used in ensemble or multimodal fusion pipelines because they connect directly to structured radiomics or clinical variables and expose feature attributions.

The number of hybrid approaches (37/371, 10%), including CNN-ML ensembles, multitask frameworks, and Concept Bottleneck Model architectures has increased between 2023 and 2024. Hybrid approaches integrate complementary model families or data sources to balance performance with interpretability. Typical patterns include CNN and ML pipelines, where a deep encoder produces compact features that a transparent learner (eg, SVM, RF, and gradient boosting) turns into decision rules or feature rankings. Overall, these designs emphasize embedding interpretability within the training pipeline rather than relying solely on post hoc explanations.

A small portion of studies (7/371, 1.9%) use modeling approaches that are less common in current practice. Notably, concept and prototype-based designs (eg, Concept Bottleneck Models, ProtoPNet, and Breast Imaging Reporting and Data System [BI-RADS]-NET) and architectures with explicit causal mechanisms (eg, Causeg-Net and BDStable-Net) aim to anchor model learning to clinically meaningful factors and human-understandable concepts, providing built-in transparency.

A qualitative analysis of the corpus identified 6 recurring categories in terminology (Figure S3 in Multimedia Appendix 4). Most usage is concentrated in the categories of interpretable or interpretability and explainable or explainability. Trustworthy and transparent also appear, but as secondary themes. Other labels, such as reliable or visualization, form a “long tail.” Because the key terms for retrieval were both explainable and interpretable, these observations are descriptive rather than unbiased frequency counts. The takeaway is that xAI vocabulary is diverse and often used interchangeably in literature. Therefore, we report this mapping to support replication and facilitate future searches.

Analysis across 371 studies published from 2017 to 2024 showed that post hoc xAI methods were dominant, used in 82.2% (n=305) of all studies (Figure S4 in Multimedia Appendix 4 and Table 5). Most of these studies used post hoc as the primary means to add transparency to their predictive models. Studies using inherently transparent models, such as those with built-in structures (eg, DTs and linear coefficients) that enable transparency by design, accounted only for 5.7% (n=21) of studies. Hybrid methods accounted for 12.1% (n=45) of studies, combining inherently interpretable models with post hoc techniques to provide more robust or complementary explanations.

Post hoc approaches dominated (n=305, 82.2%), followed by hybrid (n=45, 12.1%) and ante hoc (intrinsic) methods (n=21, 5.7%). Within the post hoc group, visualization-based explanations, such as saliency maps and activation heatmaps, were the most prevalent (n=163, 53.4%), followed by feature relevance approaches including SHAP and layer-wise relevance propagation (n=111, 36.4%). Combination (multimodal) post hoc strategies accounted for 4.3% (n=13). Other categories, such as text explanations (n=7, 2.3%), explanations by example (n=6, 2%), and explanations by simplification (n=5, 1.6%), were comparatively infrequent, underscoring the dominance of visual and feature-based interpretability techniques in current xAI research.

The temporal evolution of explainability methods from 2017 to 2024 reveals a clear and notable trend. Post hoc methods have consistently remained the dominant approach; however, the composition of the methods used has changed. Whereas visualization-based methods, such as Grad-CAM, were previously most common, they are now increasingly rivaled by more recently available feature-perturbation methods, such as SHAP, which have even become dominant in some years.

The extent to which the study authors clearly defined the goals for explainability varied throughout the study period. Early studies were more likely to define explainability goals broadly (eg, “to improve interpretability” and “to provide explanations”) without specifying the particular clinical applications or potential user groups. In contrast, more recent studies increasingly define specific objectives (eg, to foster clinician confidence, to make model output consistent with diagnostic standards, such as BI-RADS, and to enable visual validation of lesion location).

In only 21 of 371 studies (approximately 5.7%), researchers used models designed to be interpretable from the beginning, rather than explaining them retrospectively, indicating that fully intrinsic interpretability remains rare.

The largest group of studies (9/21) achieved interpretability by imposing structural or architectural constraints within deep models. Examples include the hierarchical semantic CNN or hierarchical semantic network for lung nodule malignancy [44], Thyroid Imaging Reporting and Data System–based multitask networks for thyroid nodules [45], prototype-based CNNs for mammography [46], and lesion-level risk aggregation models for metastatic lung cancer prognosis [47]. In these cases, the model provides clinically relevant information, such as nodule shape or margin [48], assigns specific per-lesion risk scores [47], or presents prototypical patterns as part of its standard inference process [46].

In a second subset (3/21), studies used generalized additive or sparse linear models, such as Explainable Boosting Machines and least absolute shrinkage and selection operator–regularized linear or logistic regression, for prognosis of treatment response (local failure in head and neck cancer [49]; 3-year overall survival in early-stage non–small cell lung cancer [50]) and prediction of tumor markers in kidney cancer [51]. Feature-wise contribution functions and sparse coefficients on clinically meaningful features serve as the built-in explanations of these models.

In total, 5 studies (5/21) used transparent, feature-engineered pipelines, for example, for prostate cancer segmentation [52], or liver cancer treatment monitoring [53], and tumor classification in breast MRI [54], focusing on explicit feature rankings. Furthermore, 2 studies defined their interpretability in terms of biologically grounded phenotypes, such as radiomic factors linked to CD8+ T lymphocyte infiltration and tumor microenvironment phenotypes, using these as model inputs instead of opaque latent features [55,56].

Moreover, 2 other studies (2/21) were based on probabilistic and fuzzy models (fuzzy classification for radiotherapy toxicity and Bayesian multitask learning regression), providing global interpretability through human-readable fuzzy rules or task-specific linear mappings between predictors and clinical targets. Rule-based and symbolic learners were also present in this xAI category (2/21; rule-fit for chemoradiation PET response in non–small cell lung cancer [57] and grammatical evolution classifiers for breast cancer [58]). They generated explicit human-readable rules and formulas.

Studies in this category generally adopted transparent models to meet clinical interpretability requirements, such as providing explicit feature importance rankings from radiomics features or feature-outcome correlation plots, rather than relying on visual explanations.

In this review, we examined how studies validated their explainability components, the metrics and strategies adopted, and the extent of user involvement. Among all included studies, 193 (52%) reported at least 1 form of xAI validation. Because some studies used multiple validation strategies, categories overlap, and totals can exceed 193. Expert or user-based validation was reported in 104 studies (28% of all studies; 53.9% of validated studies). Mixed methods validation was reported in 74 studies (19.9% of all studies; 38.3% of validated studies). Quantitative metrics were reported in 10 studies (2.7% of all studies; 5.2% of validated studies), and domain or clinical-knowledge validation in 8 studies (2.2% of all studies; 4.1% of validated studies).

The majority of studies (n=104/193, 53%) based their validation primarily on expert feedback obtained from doctors, radiologists, or clinicians. They assessed whether the explanations aligned with medical experts’ reasoning during diagnosis. This was achieved through interviews with them, surveys of readers, or usability tests to determine if the AI explanations were logical and understandable. Even when researchers did not explicitly label their studies as “validation of xAI,” many included some form of evaluation of the usability and reasoning of the AI’s explanations, contributing to the overall objective of validating the AI’s explanatory ability.

Fewer studies (10/193, approximately 5%) relied on indirect assessment of explanation quality using performance metrics or model comparisons. In particular, studies reported using various metrics to measure the quality of AI-generated explanations. Among the most common metrics were area under the curve (AUC), precision-recall AUC, and feature ranking consistencies. However, most of the time, these metrics assess the quality of the model’s predictions rather than the explanations themselves.

Some studies validated explanations by checking their consistency with current clinical and/or biological knowledge (8/193, 4%). For example, they verified agreement between the model’s predictions and histopathological characteristics; compared the model’s explanatory features with radiologists’ accepted standards to ensure consistency; or tested whether the explanatory features remained consistent across all data partitions. All these methods help demonstrate the validity of the clinical application of the xAI system in medicine.

In addition to the above types of validation, some studies (74/193, 37.7%) combined 2 or more methods to validate the quality of AI explanations. Most often, this mixed methods approach involved combining expert feedback with one of the 3 types of quantitative validation described previously. By doing so, the authors aimed to balance the need for objectivity in explanation quality with the need to understand experts’ perspectives on trustworthiness and usability.

For example, studies such as the one by Nowakowska et al [101] used SHAP-based feature importance analysis alongside expert radiologist review to validate the clinical coherence of model explanations. Finally, another study showed that AUC-based model performance can be triangulated with reader-study feedback to validate the alignment of visual explanations with diagnostic reasoning [48,102].

Based on the review of 371 studies, most of them did not share their code (Figure 3). Of the 371 reviewed studies, only 17.5% (65/371) reported sharing their full or partial implementation via a platform (such as GitHub) or upon request. In contrast, 82.5% (306/371) provided no access to their source code, a figure that includes studies where no code availability information was reported, which were confirmed through subset sampling to represent a lack of shared implementation. The failure to make source code available is a major hindrance to reproducing any findings, preventing peers from verifying the work and limiting advancements through collaboration in the field.

Despite the overall low transparency, a longitudinal analysis reveals a positive shift in open science practices. From 2017 to 2023, only 15.8% (27/171) of studies provided public or partial access to their code. However, among studies published in 2024, this proportion rose to 19% (38/200). While the volume of “no” results is increasing in absolute terms due to the explosion of xAI research, the relative rate of code sharing is showing a steady upward trend.

Out of 371 studies, only 45 (12.1%) reported full or partial DSS integration, while 321 (86.5%) did not, and 5 (1.3%) were unspecified (Figure 4). A temporal comparison shows a modest increase in reported integration; only 11 out of 171 studies (6.4%) from 2017 to 2023 mentioned DSS use, compared with 34 out of 200 (17%) in 2024. Among all the 2024 studies, the majority discussed potential or planned integration (69/200, 34.5%), often in the form of prototypes, user interfaces, or frameworks intended for future translation, while fewer describe actual deployment within clinical workflows (34/200, 17%).

Consistent with previous reviews in medical imaging, the dominance of post hoc explainability methods, including Grad-CAM and SHAP, reflects a trend similar to that observed in earlier literature [104,105]. This dominance is largely due to the relative ease of implementation and versatility of visual attribution methods, as well as their compatibility with a wide range of model architectures, particularly CNNs [106,107]. However, these methods generally provide qualitative explanations, which can contain a subjective side and be significantly affected by perturbation [104,108].

Although the majority of studies reporting xAI validation used some form of expert feedback, these assessments were typically limited to plausibility judgments, asking whether explanations appeared reasonable, rather than a systematic evaluation of whether model attributions corresponded to established clinical or biological knowledge.

The widespread use of SHAP across studies, driven by its ease of access and model-agnostic applicability, should be interpreted cautiously when evaluating clinical confidence. Fewer than one-third of the studies that employed SHAP included expert-based validation to assess alignment with clinical reasoning. This is notable given evidence that SHAP-derived attributions may have limited fidelity to actual model behavior [104], suggesting that technical interpretability should not be automatically equated with clinical transparency or alignment with radiological cognition.

As noted in our results, there appears to be limited adoption of inherently interpretable models, such as Explainable Boosting Machine or generalized additive model, which may be attributed to the perception—and empirical documentation—of an accuracy-interpretability tradeoff, with interpretable models showing a consistent 5%‐7% AUC penalty compared with black box approaches [104]. At the same time, the emergence of hybrid models that combine DL with traditional ML classifiers suggests growing interest in architectures that offer improved transparency while maintaining high model performance. This progression reflects trends observed in other recent surveys of explainable medical AI [109].

The stated purposes of explainability in the reviewed studies became more specific over time, shifting from general descriptions to articulating concrete clinical objectives. This trend may reflect the field’s maturation from demonstrating technical feasibility to addressing how explanations should be designed for clinical utility.

We identified CT and MRI as the two most widely used modalities in our study, although the rate at which xAI methods are validated and their application varies greatly depending on the modality. In the modality-specific subset analysis, we found that ultrasound and mammography studies reported validation of xAI more frequently than CT and MRI studies. There were also differences in method preference among the modalities. For example, mammography-based studies used Grad-CAM more often, while CT-based studies used SHAP more frequently. These differences in modality-specific application suggest that xAI may not be universally applied. Instead, xAI design and validation procedures will likely need to be tailored to each specific modality and workflow [110,111].

While approximately half of the reviewed studies reported some form of explainability validation, the approaches and rigor varied considerably. Expert-based validation predominated, typically involving radiologists or clinicians assessing whether explanations appeared plausible or aligned with their diagnostic reasoning. However, very few studies implemented deeper feedback loops—whether from expert critique or from xAI evaluation outcomes—to iteratively refine model features or architectures. This pattern suggests that validation often serves as post hoc verification rather than as an integral component of model development.

Quantitative validation methods, although less common, were typically indirect—relying on performance metrics (AUC and accuracy) that assess prediction quality rather than explanation fidelity. Few studies used formal metrics specifically designed to evaluate explanation quality, consistency, or clinical utility. This methodological gap, combined with the lack of standardized frameworks for assessing explanations, leaves most validation efforts qualitative and subjective [110].

Several limitations emerged across the reviewed literature. Many studies acknowledged that their models lacked multicenter or external validation, limiting confidence in both predictions and explanations. Training on small or imbalanced datasets has been associated with instability in both model predictions [112,113] and attribution patterns [104]. Researchers also noted that heatmap interpretations remain subjective and may not correspond to clinical reasoning, risking misinterpretation by users lacking appropriate context [104]. Additionally, the complexity or opacity of some model architectures can render generated explanations less actionable in clinical settings.

Importantly, validation results were rarely used to modify model design. Explainability was typically incorporated after model development and validation were complete, a pattern consistent with the predominance of post hoc rather than in-model approaches observed across the broader medical AI literature [114,115]. This post hoc approach limits opportunities for iterative refinement and may contribute to overestimation of clinical readiness.