This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.
Significant efforts have been made to develop artificial intelligence (AI) solutions for health care improvement. Despite the enthusiasm, health care professionals still struggle to implement AI in their daily practice.
This paper aims to identify the implementation frameworks used to understand the application of AI in health care practice.
A scoping review was conducted using the Cochrane, Evidence Based Medicine Reviews, Embase, MEDLINE, and PsycINFO databases to identify publications that reported frameworks, models, and theories concerning AI implementation in health care. This review focused on studies published in English and investigating AI implementation in health care since 2000. A total of 2541 unique publications were retrieved from the databases and screened on titles and abstracts by 2 independent reviewers. Selected articles were thematically analyzed against the Nilsen taxonomy of implementation frameworks, and the Greenhalgh framework for the nonadoption, abandonment, scale-up, spread, and sustainability (NASSS) of health care technologies.
In total, 7 articles met all eligibility criteria for inclusion in the review, and 2 articles included formal frameworks that directly addressed AI implementation, whereas the other articles provided limited descriptions of elements influencing implementation. Collectively, the 7 articles identified elements that aligned with all the NASSS domains, but no single article comprehensively considered the factors known to influence technology implementation. New domains were identified, including dependency on data input and existing processes, shared decision-making, the role of human oversight, and ethics of population impact and inequality, suggesting that existing frameworks do not fully consider the unique needs of AI implementation.
This literature review demonstrates that understanding how to implement AI in health care practice is still in its early stages of development. Our findings suggest that further research is needed to provide the knowledge necessary to develop implementation frameworks to guide the future implementation of AI in clinical practice and highlight the opportunity to draw on existing knowledge from the field of implementation science.
Artificial intelligence (AI) can potentially transform health care data into meaningful and actionable insights [
The potential benefits to patients from new health technologies are often missed owing to slow and variable uptake in practice [
Importantly, for the development of AI technologies, implementation science has evidenced that passive approaches to the dissemination and diffusion of health care technologies are rarely effective [
If the value of AI technologies is to be realized in practice, it is important to develop evidence-based approaches to AI implementation. Although it is likely that generalizable implementation theories, models, and frameworks will be able to provide valuable guidance for the implementation of AI technologies, it is likely that the nature of AI features will add new layers of complexity and pose additional challenges to effective implementation [
This study aims to explore the current state of academic knowledge of AI implementation by assessing any implementation theories, frameworks, or models that are specific to AI translation into health care practice. The study objectives are to assess the following:
What, if any, AI-specific implementation frameworks in health care exist?
How do these AI-specific implementation frameworks draw on and compare to more generalized implementation frameworks for health technologies?
What do any AI-specific implementation frameworks reveal about the challenges of AI implementation?
An interpretative scoping review was considered the most appropriate method to answer the research questions, as it provides a systematic synthesis of knowledge within a defined area, and with the aim of exploring and mapping key concepts, available evidence, and shortcomings in existing research [
The operational definitions for the term implementation, framework, and AI are listed in
Operational definitions for key concepts.
Term | Operational definition | Examples in health care |
Implementation | An intentional effort designed to change or adapt or uptake interventions into routines [ |
Adoption of heart failure prediction software Change the clinical decision support system |
Artificial intelligence | A general purpose technology based on a core set of capabilities and computational algorithms designed to mimic human cognitive functions to analyze complex data [ |
Machine learning for mortality prediction Unstructured image data analysis for radiology |
Framework | A simplification structure, overview, system or plan of multiple descriptive categories or elements (ie, constructs, concepts, and variable) that streamline the interpretation of a phenomenon [ |
NASSSa framework (for health and care technologies) [ SHIFTb evidence [ |
aNASSS: nonadoption, abandonment, scale-up, spread, and sustainability.
bSHIFT: successful health care improvement from translating evidence.
A systematic search of MEDLINE, Embase, EBM
PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flow diagram showing the review process. AI: artificial intelligence; EBM: Evidence Based Medicine.
Studies were eligible for inclusion if they were written in English and referred to the implementation of AI (eg, ML) in health care settings. All study designs or publication types were eligible for inclusion to identify the presence of implementation frameworks to guide the use of AI in health care. Conference abstracts, editorials, and technical reports were excluded. Other reasons for exclusion included studies that did not focus on AI, had no explicit focus on implementation in relation to implementation frameworks, models, or theories of AI, that were not focused on the health care setting, or were dedicated to nonhuman aspects (eg, animal health). Titles and abstracts were screened for inclusion by 2 independent reviewers (FG and DT) using the Rayyan web platform. Disagreements were resolved by consensus, and when necessary, a third reviewer was involved (JB). The agreement score during screening was substantial (κ score>0.8).
A 4-step process was used for data extraction and analysis according to the analytical framework of Arksey et al [
Second, as this field of research is relatively new, we considered it important to perform a quality assessment of the included articles, even if this is not typical for scoping reviews [
Third, to thematically analyze the literature, the frameworks used in the included studies were categorized in accordance with the Nilsen taxonomy of 5 categories of implementation frameworks [
Fourth, to analyze the literature in relation to the extent to which these findings draw on existing implementation frameworks, a deductive thematic analysis was carried out. This entailed coding the included studies deductively according to the nonadoption, abandonment, scale-up, spread, and sustainability (NASSS) framework for health technologies [
The initial search returned 2541 unique articles. We screened all abstracts and eliminated 98.86% (2512/2541) of the papers based on the exclusion criteria. The term implementation framework was widely dispersed across different areas of health care, which resulted in a high number of nonrelated papers. After abstract screening, 29 articles were subjected to full-text review, of which 76% (n=22) of them did not meet the inclusion criteria. Finally, 24% (7/29) of articles were included.
Out of 7 articles, only 2 (29%) of the articles included formal frameworks that directly addressed AI implementation in health care. Owing to the limited number of articles fulfilling the inclusion criteria, articles that partially met the criteria were also included: 14% (1/7) of the articles that included a formal framework addressing issues of ethics and AI (a topic of high importance to real-world AI implementation), and 57% (4/7) of the articles that included descriptions of elements influencing implementation (focusing on physician opinion, patient opinion, and factors influencing the use of AI in emergency departments and surgical settings).
The articles presented heterogeneous study designs. In total, 57% (4/7) of these studies were literature reviews, 29% (2/7) used a qualitative approach, and 71% (1/14) used quantitative survey data. The area of practice and the focus of the selected articles were dispersed. The papers focused on the perceptions of patients and clinicians of AI (n=2), decision-making and decision support systems (n=2), ethical and trustworthy aspects (n=1), benefits and challenges in implementing AI (n=1), and implementation elements to guide AI adoption (n=1). Further characteristics of the included studies are shown in
Characteristics of included papers (n=7).
Study | Country | Study design | Area of practice | Target population | Study focus | Study aims |
Beil et al [ |
Germany | Literature review | Intensive care | N/Aa | Ethical and trustworthy aspects in intensive care | Discuss ethical considerations about AIb for prognostication in intensive care. |
Diprose et al [ |
New Zealand | Quantitative study | Primary care | Physicians (n=170) | Perceptions of clinicians to understanding, explain and trust on AI results | Investigate the association between physician understanding of AI outputs, their ability to explain these to patients, and their willingness to trust the AI outputs. |
Fernandes et al [ |
Portugal | Literature review | Emergency department | N/A | Intelligent CDSSc for triage | Assess how intelligent CDSS for triage have been contributing to the improvement of quality of care in the EDd as well as to identify the challenges they have been facing regarding implementation. |
Loftus et al [ |
United States | Literature review | Operation room | Surgeons | Decision-making in surgeries | Propose that AI models would obviate these weaknesses and be integrated with bedside assessment to augment surgical decision-making. |
Nelson et al [ |
United States | Qualitative study | Dermatology clinics | Patients from general dermatology clinics (n=48) | Perception of patients on AI related to skin cancer screening | Explore how patients conceptualize AI and perceive the use of AI for skin cancer screening. |
Ngiam et al [ |
Singapore | Literature review | Health care | N/A | Benefits and challenges of AI in oncology | Discuss some of the benefits and challenges of big data and machine learning in health care. |
Truong et al [ |
Canada | Qualitative study | Health care | Subject-matter experts in health care (n=8) | Implementation elements to guide AI adoption | Creating an implementation framework to help health care organizations understand the key considerations and guide implementation efforts for AI. |
aN/A: not applicable.
bAI: artificial intelligence.
cCDSS: clinical decision support system.
dED: emergency department.
Overall, the quality assessment indicated that 43% (3/7) of articles [
The analysis identified the use of three primary framework categories: determinant framework, process model, and evaluation framework. Classic theories and implementation theories have not yet been identified. Across the 7 articles, the data analysis indicated that 3 (43%) articles included explicit frameworks: determinant frameworks [
The data analysis identified the presence of implementation elements in all 7 of the NASSS domains (
Among the individual articles, Nelson et al [
In total, 7 new subdomains were identified that did not explicitly fit in the NASSS framework (
Descriptions of the frameworks and framework elements in the included articles (n=7).
Study | Explicit framework? | Types of frameworka and purpose | Framework elements (stages, determinants, or aspects) | Clarity of element descriptionb | Referenced guidance or literature for framework development |
Beil et al [ |
Yes | Evaluation framework; ethical AIc | Beneficence, nonmaleficence, justice, autonomy, explicability, medical perspective, technical requirements, patient- or family-centered, and system-centered | Partial | European Commission guideline |
Diprose et al [ |
No | N/Ad; elements describe physician opinion of AI | Physician understanding and intended physician behavior, explainability, preferred to explainability methods | Partial | Absent |
Fernandes et al [ |
No | N/A; elements describe limitations to develop and implementing AI in EDe triage | Availability of data, the subjectivity of the system, methodologies and modeling techniques, validation, and geography (data from the same geographic area) | Partial | Absent |
Loftus et al [ |
No | N/A; elements describe challenges and potential of AI in surgical decision-making | Challenges in surgical decision-making (complexity, values and emotions, time constraints and uncertainty, heuristics and bias), traditional predictive analytics and clinical decision support (decision aids and prognostic scoring systems), AI predictive analytics and augmented decision-making (machine learning, deep learning, and reinforcement learning), implementation (automated electronic health record data, mobile device outputs, and human intuition), challenges to adoption (safety and monitoring, data standardization and technology infrastructure, interpretability, and ethical challenges) | Partial | Absent |
Nelson et al [ |
No | N/A; elements describe patient opinion of AI | AI concept, AI benefits, AI risks, AI strengths, AI weaknesses,f AI implementation (symbiosis, credibility, diagnostic tool, setting, and integration into electronic health records. Challenges include malpractice, misunderstanding of AI, and regulations), response to conflict between human and AI clinical decision-making, responsibility for AI accuracy, responsibility for AI data privacy, AI recommendation | Limited | Absent |
Ngiam et al [ |
Yes | Process model; AI development and implementation | Clinical problem definition or redefinition, data extraction selection, and refining, data analysis and validation, human-machine interaction, paper trial, prospective clinical trial, medical device registration, and clinical deployment | Explicit | Absent |
Truong et al [ |
Yes | Determinant framework; AI implementation | Data quality and quantity, trust, ethics, readiness for change, expertise, buy-in (value creation), regulatory strategy, scalability and evaluation | Explicit | Absent |
aType of framework according to the Nilsen taxonomy [
bExplicit: explicit definition; partial: some discussion, but no explicit definition; limited: only listed construct names, but no definition or discussion is provided.
cAI: artificial intelligence.
dN/A: not applicable.
eED: emergency department.
fOnly categories associated with artificial intelligence implementation are shown in full.
A comparison of elements identified in literature with the nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability (NASSS) framework domains. (n=7).
Condition | Technology | Value proposition | Adopters | Organization | Wider system | Embedding and adaptation over time | |||||||
|
|||||||||||||
|
Nature of conditiona (n=0) Comorbidities, sociocultural influences (n=0) |
Material and features of technology (n=7) Types of data generated (n=7) Knowledge needed to use (n=5) Technology supply model (n=2) |
Supply-side value (to developer; n=2) Demand-side value (to patient; n=2) |
Staff (role and identity; n=6) Patient (simple vs complex input; n=3) Carer (available, nature of input; n=2) |
Capacity to innovate (n=1) Readiness for change (n=2) Nature of adoption or funding decision (n=1) Extent of change to new routines (n=1) Work needed to implement change (n=2) |
Political or policy (n=2) Regulatory or legal (n=5) Professional (n=2) Sociocultural (n=2) |
Scope for adaptation over time (n=1) Organizational resilience (n=1) |
||||||
|
|||||||||||||
|
Not identified |
Types of data inputted (n=3) Dependence on other local processes and practices (n=2) Evaluation of effectiveness (n=3) |
Demand-side value (to population; n=1) |
Shared decision-making (n=3) |
Not identified |
Ethics (population equity or discrimination; n=2) Role of human oversightb (n=3) |
Not identified |
aThese elements were not explicitly mentioned in the framework or list of elements, but they were considered in the manuscript (nature of condition, 6 articles; comorbidities and sociocultural influences, 2 articles).
bCan be considered across multiple domains.
This literature review demonstrates that understanding how to implement AI in health care practice is still in its early stages of development. Although our study search terms identified a large number of articles, only 7 articles were included in the final analysis. Only 29% (2/7) of these articles included formal frameworks that directly addressed AI implementation in health care, and the other 71% (5/7) of the articles provided descriptions of elements influencing such implementation.
The importance of developing knowledge of how to implement AI in health care was highlighted in many of the rejected articles, but despite acknowledging this, the articles provided little or no substance to support their claims, or guidance on how to move forward. A challenge to building knowledge in this field was underscored during the screening process, where many articles mentioned AI but were excluded because they focused on health care technologies unrelated to AI; for example, eHealth and telemedicine. The inappropriate labeling of technologies as AI likely reflects the hype surrounding the AI concept and the tendency to adopt fashionable terms to increase attention, readership, and likeliness of publication [
Given the recognition of the importance and challenges of AI implementation [
To understand the overlap of concerns between AI-specific implementation challenges and more general health technologies, we mapped the elements listed in the 7 papers against the NASSS framework, which was specifically developed to guide the uptake of health technologies [
A small number of elements identified in the 7 papers did not align with the existing NASSS subdomains. As such, we propose 7 new subdomains that can supplement the NASSS framework. Of the newly identified subdomains, some highlight issues that are likely relevant to all forms of health technologies and may have specific implications within AI. For example, the newly identified subdomain of
The subdomain
Other newly identified subdomains appear to be more highly relevant or potentially problematic for AI than for other forms of health technology. For example, understanding the
Our findings highlight the need to develop an AI-specific implementation framework, drawing on empirical research related to AI implementation efforts, and drawing on existing knowledge and experience within the implementation science community. There is a great, currently unrealized, opportunity to draw on insights from the implementation of science literature to enhance and accelerate the implementation of AI. There is no need to repeat the mistakes or reproduce learning that has already been achieved, and there is an opportunity to use the theoretical and practical insights from others to provide an evidence-based foundation that can accelerate the implementation of AI in health care.
Our findings suggest that the implementation of AI is viewed through a narrow lens, focusing on the design of the technology and its interaction with the immediate user. Lessons from implementation science suggest the need to extend attention to understand how the technology will influence and interact with the context in which it is implemented, including understanding existing processes and practices of care within each local setting, and how systems work at micro, meso, and macro levels to support or hinder technology uptake. Such insights can only be gained from active engagement of relevant stakeholders, frontline staff, patients, and careers, their engagement is essential to understand how new technologies that are based on AI will be received, and how trust can be built to ensure that design is centered on the needs and practical constraints and requirements of the health care system (ie, to produce useful, trusted, relevant, and actionable knowledge). Lessons from implementation science suggest that obtaining this knowledge, using it to inform technology design, and addressing wider implementation issues is time intensive and reliant on good quality relationships between diverse and often conflicting groups of stakeholders. However, time and time again the implementation literature demonstrates the necessity of this work for successful implementation. Harnessing such insights could provide guidance to health care professionals responsible for implementing AI in practice, for decision makers and policy makers to ensure effective implementation plans are in place, and for the AI designers and promotors who need to be aware of the implications of real-world deployment to ensure that AI products are suitable for implementation.
Any AI implementation framework also needs to recognize the heightened and perhaps unique needs and challenges of introducing AI in health care, including meaningful decision support, ethical dilemmas (privacy and consent), transparency, effectiveness, interpretability, and establishing trust in
Although this study was conducted in a structured and systematic manner, only a small number of papers met the inclusion criteria, and these papers were of rather low quality in terms of methodological clarity and rigor, and in the clarity of descriptions and definitions of elements influencing AI implementation. In addition, the included studies that were based on empirical data were conducted only in 3 high-income countries, limiting the generalization of the findings to other contexts. For example, the use of AI in health care in low-income countries is still nascent, and therefore some subdomains of the NASSS framework might be irregularly advanced in this context (eg, legal, regulatory, and social cultural). These characteristics emphasize the importance of reflecting on the findings of parsimony. Together, these limit the reliability and generalizability of the cumulative findings from our analysis and highlight a gap in the literature that requires further empirical and theoretical research.
We chose to use the NASSS framework for deductive analysis of the included papers as it represents one of the most advanced frameworks dedicated to understanding implementation of health care technologies and was informed by extensive empirical research and literature review [
Scoping reviews often search for the identification and conceptualization of complex, emergent, or ill-defined concepts. Unlike traditional systematic reviews guided by well-defined constructs, it may be unfeasible to screen and synthesize all relevant literature on an emergent topic [
This literature review demonstrates that the research literature on AI implementation in health care lacks theoretical development and is poorly connected to existing implementation frameworks or models developed within implementation science. This means that potential specific challenges around AI implementation are largely unrevealed, and that further empirically based research is needed to provide the knowledge necessary to develop implementation frameworks to guide future implementation of AI in clinical practice.
Health care database search syntax.
Quality appraisal of the selected papers.
Data analysis matrix.
artificial intelligence
Critical Appraisal Skills Programme
Medical Subject Headings
nonadoption, abandonment, scale-up, spread, and sustainability
Preferred Reporting Items for Systematic Reviews and Meta-Analyses
The funders for this study are the Swedish Government Innovation Agency Vinnova (grant 2019-04526) and the Knowledge Foundation (grant 20200208 01H). The funders were not involved in any aspects of study design, collection, analysis, interpretation of data, or in the writing or publication process.
All authors (FG, DT, JN, JR, JB, and PS) made significant contributions to the original paper. All the authors contributed to the study design. Applications for funding and coproduction agreements were put in place by PS and JN. FG, DT, and JB identified and selected relevant studies, FG and DT charted the data, and FG, DT, JN, JR, and PS analyzed, summarized, and reported the results. The manuscript was drafted by FG, DT, JN, JR, and PS, and all authors critically revised the paper in terms of important intellectual content. All authors have read and approved the final submitted version.
None declared.