The rapid development of artificial intelligence (AI) has demonstrated profound impacts across various industries, particularly in the health care sector. The application of AI has shown significant potential and is widely used in areas such as disease diagnosis, patient monitoring, robotic surgery, and clinical decision-making [1]. However, with the increasing prevalence of AI technology, issues concerning doctors’ acceptance, trust, and willingness to use AI have garnered widespread attention. A study revealed that only 10% to 30% of doctors use AI in real-world scenarios [2]. The poor acceptance and low use of AI systems by users are influenced by various factors, including the technical characteristics of AI itself, individual factors (eg, users’ AI literacy), organizational factors (eg, advocacy by management), and policy-related issues (eg, responsibility attribution in the use of AI). These challenges hinder the widespread adoption of AI technology [3-7].

Among the technical characteristics of AI, explainability and integrability are considered 2 key factors influencing doctors’ acceptance and use of AI [8,9]. While other elements, such as security and social influence, also play important roles in clinicians’ trust in AI [10,11], their practical impact on daily clinical adoption differs. Security concerns are essential for AI implementation, but are most often managed at the technical or regulatory level [10]. Social influence, encompassing peer and organizational advocacy, tends to influence adoption at the institutional level [12]. In contrast, multiple systematic reviews and clinician surveys have identified explainability and integrability as the most immediate and actionable factors influencing real-world AI adoption in health care [8-10,13-16]. Accordingly, focusing on explainability and integrability provides practical, user-centered insights for promoting effective integration of AI into clinical practice. In this study, explainability refers to the extent to which an AI system provides human-understandable and faithful representations of its decision-making process [13,14], while integrability refers to the extent to which AI systems can be embedded into clinical workflows with minimal disruption, ensuring usability, interoperability, and alignment with routine practices [10,15,16]. Importantly, integrability is a broader concept that encompasses interoperability. While interoperability ensures the technical capacity for systems to exchange and interpret data using standardized formats, integrability goes further to consider how AI systems align with clinical roles, decision-making contexts, workflow timing, and user experience [17,18]. A system may be technically interoperable but still lack integrability if it fails to deliver value in practice or imposes additional burdens on clinicians.

Explainability is crucial in the medical field, where decision-making is highly complex and involves significant risks. Clinicians need to ensure the accuracy and safety of AI outputs before they can trust and rely on AI. A lack of explainability in AI clinical decision support systems (AI-CDSS) may lead to distrust among decision makers and reduce their willingness to use these technologies [19]. In addition, integrability ensures that AI-CDSS can seamlessly integrate into existing clinical workflows. When AI-CDSS are effectively embedded into doctors’ daily routines, clinicians can more efficiently access and use patient data, receive recommendations that better meet practical needs, and reduce the time spent on redundant data entry, allowing them to focus on clinical decision-making [20]. Conversely, a lack of integrability in AI-CDSS may negatively impact clinician work by increasing operational complexity, workload, and time costs. Such systems, which fail to align with practical requirements, can reduce doctors’ willingness to adopt them [21-23].

However, on the one hand, most existing studies primarily neglect the end users, namely, physicians’ understanding of and need for explainable AI [24-27]. Studies have mainly concentrated on the perspectives of developers and researchers and have developed methods to present technical descriptions of model processes in AI [27], although such approaches seem meaningless if they are not aligned with physicians’ understanding of and requirement for explainability in real-world application in medical settings [26]. On the other hand, a few studies have explored how system compatibility, user-friendly design, and workflow adaptability [24-26] may contribute to the seamless integration of AI into the clinical workflow. However, what AI integrability entails and what its underlying components are from the physicians’ perspective remains unclear.

Therefore, this study aims to systematically review existing literature on explainability and integrability of AI systems in health care from the perspective of health care professionals (HCPs). Specifically, it seeks to identify how these 2 factors—explainability and integrability—influence clinicians’ acceptance and use of AI-based decision support systems. On the basis of the findings, the study proposes a conceptual framework that synthesizes key user-centered concerns, with the goal of informing the design of more acceptable and adoptable clinical AI tools.

We conducted this systematic review of systematic reviews according to a protocol registered in PROSPERO (CRD420250652253), and the databases we searched included PubMed, Web of Science, Scopus, IEEE Xplore, ACM Digital Library, and arXiv. We did not explicitly search proceedings from major machine learning and AI conferences such as Association for Computational Linguistics, Neural Information Processing Systems, International Conference on Learning Representations, or International Conference on Machine Learning, as many papers from these venues are concurrently available on arXiv. We used keywords such as “Explainable AI,” “explainability,” “XAI,” “AI integrability,” “usability,” and “human-centered AI” (the detailed search strategy is provided in Multimedia Appendix 1) and covered publications from January 2014 to July 2024. In addition, references cited in the retrieved articles and reviews were manually screened.

Studies were eligible for inclusion if they addressed either the explainability or the integrability of AI in health care (ie, meeting at least one of the following two sets of topic-specific criteria): (1) explainability, the study must address at least 1 of the following aspects—how AI processes input information; how conclusions are reached; the rationale behind these conclusions; how explainability fosters user trust and their use; or the knowledge, attitudes, and perceptions of HCPs regarding AI explainability—and (2) integrability, the study must address at least 1 of the following aspects—how AI integrates with hospital information systems and clinical workflows; how integrability fosters HCPs’ trust and use; or the knowledge, attitudes, and perceptions of HCPs regarding AI integrability.

In addition, studies were required to (3) focus on the perspective of AI end users, specifically HCPs such as doctors, nurses, and medical laboratory staff and (4) be original research, including original quantitative, qualitative, or mixed methods studies.

Editorials, reviews, conference abstracts, narrative studies, and studies focusing on the perspectives of AI developers, engineers, or technical professionals were excluded (Textbox 1).

To maximize comprehensiveness, we initially applied a broad search strategy, but studies were then screened against strict inclusion and exclusion criteria based on the study objectives. YL initially screened titles and abstracts to assess eligibility, with all decisions systematically recorded in a structured Microsoft Excel spreadsheet to ensure transparency. Owing to resource constraints, this stage was conducted by a single reviewer following strictly defined inclusion and exclusion criteria to minimize potential bias. For full-text screening, 2 reviewers (YS and CL) evaluated each study using a standardized Excel form designed to promote consistency in assessment. Any disagreements or uncertainties were discussed with a third author (JZ) until a consensus was reached, further strengthening the reliability of the screening process. Data extraction was conducted by a single reviewer but performed twice to ensure accuracy and completeness. Extracted items included study characteristics, AI system details, target health care settings, participant types, and key findings related to explainability and integrability. Any uncertainties during extraction were resolved by referring back to the full text.

To capture users’ perceptions of explainability and integrability across both health care and general domains, we initially adopted a broad search strategy without restricting to medical-specific terms. However, as 21 (95%) of the 22 included studies were from the health care domain, the final analysis focused on medical contexts.

All included studies underwent quality assessment based on their respective study designs. Quantitative and qualitative studies were evaluated using the Joanna Briggs Institute critical appraisal checklists, while mixed methods studies were assessed using the Mixed Methods Appraisal Tool (version 2018). For the JBI checklists, studies scoring ≥80% (ie, meeting at least 8 of 10 items) were considered high quality, 60% to 79% as medium quality, and <60% as low quality. For the Mixed Methods Appraisal Tool, studies that met ≥4 of 5 criteria were rated high quality, while those meeting 3 were considered medium quality. Only studies rated as medium or high quality (22/26, 85%) were included in the final synthesis.

The data extraction process included the following items for each eligible article—(1) basic information: authors, year, study region, content, and participants; (2) methodology: study design, study population, data collection methods, and data analysis methods; (3) results: HCPs’ understanding and needs regarding AI explainability or integrability.

This study analyzed quantitative and qualitative results separately and integrated them through narrative synthesis, using a descriptive method to present the findings [28]. For qualitative data, we followed the guidelines of Braun and Clarke for thematic analysis. An inductive approach was applied to code data, covering 4 stages: familiarization with the data, initial coding, identification of themes, and review of themes. Thematic analysis was conducted using both deductive and inductive approaches. For the analysis of explainability, we adopted a deductive coding structure based on a preexisting conceptual framework, which includes 3 first-order dimensions: preprocessing, model-level, and postprocessing explainability [25]. Specifically, preprocessing explainability refers to enhancing transparency during data preparation and feature engineering before model training. Model explainability focuses on understanding and interpreting the inner mechanisms, parameters, or representations of the model itself during training. Postprocessing explainability involves applying interpretability techniques after model predictions, which can provide both local (instance-level) and global (model-level) explanations of the model’s behavior. These categories guided our initial coding and interpretation. Within each of these dimensions, we generated subthemes inductively from the data to capture participants’ nuanced perspectives. For integrability, no prior framework was applied; all themes were developed inductively based on participants’ responses. Because there is a lack of theory guiding the coding for integrability, themes regarding AI integrability emerged based on the coding of extracted text from retrieved studies. NVivo 10 was used for thematic analysis. YL conducted the initial coding individually, and the coded data were then reviewed by a second person to ensure the validity of the themes. Any discrepancies in coding were discussed in group meetings, where consensus was reached.

In addition, quantitative data related to the explainability and integrability of medical AI from the user’s perspective were also extracted. Owing to substantial heterogeneity in study outcomes and exposure measures, a meta-analysis was inappropriate, and a narrative synthesis was used to analyze the collated studies, including methods, sample size, participants, outcome variables, and dimensions of explainability and integrability.

A total of 11,888 articles were retrieved through the search and their references. Among them, 26 (0.22%) articles met the inclusion criteria for this study. After quality evaluation, 4 (15%) low-quality articles were excluded, and a total of 22 (85%) articles were included in the analysis. The study selection process is shown in Figure 1.

All the included studies were published in 2020 or later. Most of the studies were conducted in developed countries (18/22, 82%), including 9 (41%) from the United States, and only 2 (9%) studies originated from developing countries (China and Brazil). Regarding study type, most studies (17/22, 77%) were qualitative in design, and 5 (23%) studies adopted quantitative or mixed method approaches. The basic characteristics of all the included studies are presented in Table 1.

This study included 3 quantitative and 2 mixed method studies, focusing on HCPs’ willingness to use AI, the key influencing factors, explainability needs, and integrability. Only 1 quantitative study addressed AI integrability. Owing to the limited number of quantitative studies, they primarily serve to complement and validate the qualitative analysis in this study. See Table 2 for details.

Two studies explored factors affecting physicians’ willingness to adopt AI, with a focus on explainability and ease of use. One mixed method study found that 77% of HCPs managing diabetes were willing to use AI, citing ease of use (68%) as a key factor [23]. Another study revealed that 25.6% of participants identified a lack of understanding of underlying technology as a barrier [30], which confirms the focus of HCPs on usability and explainability described in the previous section of the research.

Two studies focused on explainability needs. One found that post hoc local explanations, such as those provided by logistic regression and Shapley additive explanations (SHAP), received higher usability scores from clinicians than model-level explainability [11]. Another study found that clinicians rated diagnostic information, certainty, and related reasoning as very important, particularly when their diagnoses conflicted with AI recommendations [38]. These quantitative results support and validate the qualitative findings that post hoc local explainability is crucial for HCPs.

One study examined AI integration into workflows, comparing its placement in different stages of decision-making. It found that AI support at the start of a diagnostic session increased participants’ confidence and perceived usefulness but also highlighted that poor integration could increase task complexity and workload [44].

To enhance HCPs’ trust and use of AI-CDSS in future real-world clinical settings, this study adopted a mixed systematic review approach to synthesize evidence regarding AI explainability and integrability from the HCPs’ perspective. To the best of our knowledge, this study is the first to systematically summarize the concept of “AI integrability” from the HCPs’ perspective. It refers to the ability of AI systems to be easily and seamlessly integrated into workflows, providing timely, appropriately scaled, and practically relevant prompts or recommendations at the right points, without requiring excessive effort from the HCPs. HCPs’ needs for AI integrability are primarily reflected in 3 aspects: system compatibility, usability, and workflow adaptation.

Second, this study decodes the components of AI explainability based on HCPs’ lived experiences. It identifies that the core HCPs’ requirements of AI explainability can be divided into 3 stages, namely, data preprocessing, the AI model itself, and postprocessing explainability. Unlike the results from AI developers and researchers, the study found that general users (HCPs) are more focused on the explainability of the postprocessing stage, particularly local explainability, such as the importance of specific output features and the certainty of results. From the HCPs’ perspective, an explainable AI must clearly present data sources, processing workflows, model structures, decision mechanisms, and their rationales, using tools such as visualization and user-comprehensible language and logic to help HCPs understand and trust AI.

This study is the first to systematically review AI integrability from a comprehensive perspective. Current discussions about easily integrable AI only sporadically mention its compatibility with other systems and its ability to integrate into HCPs’ workflows (eg, the location of the AI and when it provides assistance), but lack in-depth and systematic exploration [48-50]. For instance, the study by Maleki Varnosfaderani and Forouzanfar [51] discussed the possibility of integrating AI with medical practice but did not thoroughly examine the specific needs faced by HCPs during the integration process. A study from rural clinics in China reported various tensions between AI-CDSS design and the rural clinical environment, such as misalignment with local environments and workflows, technical limitations, and usability barriers [22]. Another study concerning AI-CDSS in emergency departments identified integrability factors (eg, time, treatment processes, and mobility) through interviews with 12 emergency department doctors, but it was limited to a specific environment with a small sample size [45], resulting in poor extrapolation capability.

This study proposes, from the perspective of HCPs’ needs, that AI explainability should not only focus on technical transparency but also emphasize HCPs’ understanding and trust, particularly in clinical settings, where AI explanations should support HCPs in making more accurate and effective decisions. Existing research in explainable AI primarily focuses on algorithms, with related reviews mainly discussing taxonomies of explainability and technological innovations [25,52]. For example, Markus et al [53] proposed an explainability framework that mainly focuses on providing better tools for developers but largely explores explainability from an algorithmic perspective. Similarly, Amann et al [14] highlighted ethical and technical issues in explainable medical AI, pointing out the multidimensional nature of explainability, but their research remains focused on algorithm optimization and technical compliance. This study emphasizes the user perspective, offering more practical guidance for the design and promotion of explainable medical AI. The study found that HCPs focus more on postprocessing local explainability, meaning how specific predictions made by the model can explain changes in the patient’s condition or decision-making basis. This aligns with Shin [54], who emphasized that local explainability and causal relationships are key to user trust. A systematic review from medical and technical perspectives also supports this view [55]. Unlike data experts, who focus on model and data-layer explainability [25], users prioritize post hoc explainability. Some experimental research shows users (eg, doctors) have a higher understanding and acceptance of post hoc explanations, which are more actionable than traditional technical explanations [56,57]. This preference stems from 3 key clinical needs. First, post hoc local explanations help clinicians understand AI predictions in the context of individual patient cases, enabling more personalized and relevant decision-making [54]. Second, given their professional and legal responsibilities, doctors need to justify their choices based on understandable and traceable reasoning. Post hoc explainability provides the transparency required to assess whether AI outputs align with clinical guidelines and ethical standards [53]. Third, in high-pressure clinical environments, HCPs prioritize usability over theoretical clarity. Local, case-specific explanations are more practical and immediately applicable, which enhances trust and facilitates integration into routine workflows [58]. Thus, this study’s conclusion better matches real clinical scenarios, offering insights for developers to create AI-CDSS that meet the needs of HCPs.

Despite these advances, significant challenges remain in achieving explainability and integrability of AI-CDSS in clinical practice. These challenges are as follows.

As noted in the previous section, AI faces persistent challenges in explainability and system integration. These cannot be resolved through isolated interventions but require a structured, phased approach. The exploration, preparation, implementation, and sustainment framework—comprising exploration, preparation, implementation, and sustainment—offers a widely validated model for supporting health care technology adoption and enhancing clinician acceptance of AI systems [80]. To systematically address these challenges, we draw on the exploration, preparation, implementation, and sustainment framework, providing structured strategies for each phase of implementation.

In the exploration phase, institutions should identify clinical needs and collaborate with multidisciplinary teams (eg, physicians, nurses, and IT staff) to assess AI integration opportunities. Prioritizing explainability, especially alignment with clinical reasoning, is critical. Models supporting SHAP, LIME, or other visual local explanation tools are recommended, alongside qualitative feedback collection from end users [81].

The preparation phase focuses on resolving integration barriers and tailoring explanations for different roles. Multilevel explanation interfaces can accommodate varying expertise levels [32,57]. Close coordination with IT departments is essential to embed AI tools into existing EHR systems, minimizing manual input and workflow disruption [71,72]. Organizational readiness and role-specific training are also key to successful adoption [82].

In the implementation phase, a phased deployment strategy helps minimize disruption and support gradual clinician adaptation. AI-CDSS tools can first target low-risk, supportive tasks (eg, risk alerts and abnormal laboratory flagging), then gradually expand to core decision-making such as diagnosis or treatment support. To ensure alignment with clinical needs, implementation should combine performance metrics (eg, alert response times and override rates) with user feedback (eg, satisfaction ratings and suggestion boxes). Regular log reviews and focus groups can surface usability issues and guide iterative improvement [83].

The sustainment phase focuses on the long-term integration of AI into clinical workflows. Continuous monitoring of system performance and user experience is essential to ensure sustained adoption [84]. As models evolve, transparent update mechanisms—such as automatically generated explanation revisions and change logs—should be maintained to support clinician trust and promote continued engagement with the system.

This study offers a systematic exploration of AI integrability from the HCPs’ perspective, providing a conceptual framework to guide medical AI design and development. Unlike previous studies focusing on technical developers or researchers’ perspectives [85,86], it emphasizes the needs of actual HCPs, such as physicians. It establishes an AI explainability framework based on their priorities in data preprocessing, model structure, and postprocessing. This approach facilitates the development of user-centered AI-CDSS, promoting its acceptance and use by HCPs.

The limitation of this systematic review is the limited number of quantitative studies, which restricts quantitative analysis and statistical inference. Future research should include high-quality quantitative studies to validate and complement the conclusions.

Although we used a comprehensive set of keywords, the decision not to use truncation (eg, an asterisk) may have led to the omission of some relevant studies. This choice was made to maintain specificity, but it may have limited the search breadth. To address this, we supplemented the search with citation tracking. This limitation is acknowledged in the review to clarify the scope of our search strategy.

In addition, while emphasizing user perspectives, this review provides limited analysis of varying needs among different medical roles. This is partly because of the small number of eligible studies and the lack of detail regarding specific clinical tasks, settings, or user groups in many of the included papers. These limitations made it difficult to conduct deeper, context-sensitive analysis of HCPs’ perceptions of explainability and integrability. In this regard, we recommend that future research draw on implementation frameworks such as the Consolidated Framework for Implementation Research or Promoting Action on Research Implementation in Health Service framework to better account for the contextual and role-specific factors that shape HCPs’ experiences. These frameworks can support more nuanced analyses of clinical settings and tasks, guiding the development of AI tools that are better aligned with real-world practices.

In conclusion, the explainability and integrability of medical AI are key factors influencing its acceptance and use in clinical settings. On the basis of the user-centered conceptual framework proposed in this study, future AI design should focus on HCPs’ needs to enhance explainability and integrability, thereby promoting HCPs’ acceptance and use and improving its effectiveness in real-world clinical applications.