The growing adoption of electronic medical records (EMRs) in many high-income countries has resulted in improvements in health care delivery through the implementation of clinical decision support systems at the point of care [1]. To meet the ever-accelerating demands for clinical care, various innovative models have been developed to harness the potential of EMR data [2-4]. These new care models aim to enable health care organizations to achieve the quadruple aim of care, which includes enhancing patient experience, advancing providers’ experience, improving the health of the population, and reducing health care costs [5].

Artificial intelligence (AI) holds the potential to improve health system outcomes by enhancing clinical decision support systems [6,7]. AI aims to augment human intelligence through complicated and iterative pattern recognition, generally on large data sets that exceed human abilities [8]. While a large body of academic literature has demonstrated the efficacy of AI models in various health domains, most of these models remain as proof of concept and have never been implemented in real-world workflows [9]. This demonstrates the relatively inconsequential endeavors of many AI studies that fail to produce any meaningful impact in the real world. Even with the substantial investments made by the health industry, the implementation of AI analytics in complex clinical practice is still at an early stage [10]. In a limited number of instances, AI has been successfully implemented, largely for nonclinical uses such as service planning or trained on limited static data sets such as chest x-rays or retinal photography [11]. The factors influencing the success or failure of AI implementations in health are poorly investigated [12]. Understanding these barriers and enablers increases the likelihood of successful implementation of AI for the digital transformation of the health system [13,14], ultimately aiding in achieving the quadruple aim of health care [5].

A 3-horizon framework has been previously published to help health systems create an iterative pathway for successful digital health transformation (Figure 1 [15]). Horizon 1 aims to optimize the routine collection of patient data during every interaction with the health system. In horizon 2, the data collected during routine care are leveraged in real or near real time to create analytics. Finally, in horizon 3, the insights from data and digital innovations are collated to develop new models of care. A health care system focused on continuous improvement is referred to as a learning health system (LHS) that uses routinely collected data to monitor and enhance health care outcomes consistently [16]. When health care organizations reach the third horizon, they can leverage data in near real time to create ongoing learning iterations and enhance patient care, leading to the establishment of an LHS [17].

Regarding the 3-horizon model, EMRs are the foundation of horizon 1 (Figure 1). While many health organizations have successfully adopted EMRs into their existing workflows, the transition to horizons 2 and 3 has been challenging for many of these health care facilities [18]. A critical phase in this transition involves moving beyond the capture of EMR data for delivering analytics, including AI, aiming to improve clinical outcomes. There is little published evidence to assist health systems in making this transition [19,20].

Before conducting our review, we performed a manual search on Google Scholar using our Medical Subject Heading (MeSH) terms along with the “review” keyword to identify previous review papers that aimed at reviewing studies on the implementation of clinical AI in health care settings. We also included review papers known to our research team. Between 2020 and 2022, we identified 4 reviews that were relevant to the implementation of AI in health care systems [21-24]. Overall, these papers reviewed 189 studies between 2010 and 2022. The characteristics of these reviews, outlined in Table 1, were the year of publication, the targeted care settings, the source of data, the predictive algorithm, and whether the predictive algorithm was implemented.

The prior works identified 20 enablers and 13 barriers to AI implementation in health care across 4 categories: people, process, information, and technology (Multimedia Appendix 1 [21-24]). Overall, the findings derived from these review papers hold significant potential in providing valuable insights for health systems to navigate the path toward digital health transformation. One prevailing shortcoming of these studies is the absence of alignment with evidence-based digital health transformation principles to provide health care organizations with actionable recommendations to enable an LHS [17], therefore limiting their applicability for strategic planning within hospital organizations.

Hospitals are intricate hubs within the health care ecosystem, playing a central role in providing comprehensive medical care and acting as crucial pillars supporting the foundations of health care systems worldwide. Understanding the factors influencing the success or failure of AI in hospitals provides valuable insights to optimize the integration of these emerging technologies into hospital facilities. While the previous reviews included all health care settings [21-24], our study only focused on hospital settings. Given the limited instances regarding the implementation of AI in hospital facilities, this study explored the real-world case studies that have practically reported their AI implementation solutions in hospital facilities, aiming to synthesize the evidence of enablers and barriers within their implementation process. In addition to the inclusion of these implementation case studies, we incorporated implementation guidelines as they can potentially assist in the overall understanding of AI implementation in hospitals. This study also focused on aligning the evidence of enablers and barriers within the 3-horizon framework [15], offering a way to establish an empirical infrastructure. As a result, this can enable health care organizations to learn, adapt, and accelerate progress toward an LHS [25].

This review investigated the following research questions (RQs): (1) What enablers and barriers are identified for the successful implementation of AI with EMR data in hospitals? (RQ 1) and (2) How can the identified enablers of and barriers to AI implementation lead to actions that drive the digital transformation of hospitals? (RQ 2).

In addressing these questions, our objectives were to (1) conduct a systematic review of the literature to identify the evidence of enablers of and barriers to the real-world implementation of AI in hospital settings and (2) map the identified enablers and barriers to a 3-horizon framework to enable the successful digital health transformation of hospitals to achieve an LHS.

This study followed an extended version of the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines to outline the review methodology with comprehensive details [26]. PubMed, Scopus, Web of Science, and IEEE Xplore were searched on April 13, 2022. We reviewed prior work to determine potential MeSH keywords relevant to our study [21-24]. A research librarian helped with the definition of the MeSH keywords in PubMed and the translation of that search strategy to all platforms searched. The search strategies were applied across the 4 databases (Multimedia Appendix 2). The MeSH keywords used to search PubMed were as follows: product lifecycle management, artificial intelligence, machine learning, deep learning, natural language processing, neural networks, computer, deep learning, big data, hospital, inpatient, medical, clinic, deploy, integrate, monitor, post prediction, data drift, and regulatory. Using the Boolean operator OR, their synonyms were joined to form search phrases. Combining search phrases using the AND operator produced the final search string. We incorporated the term “data drift” to the title and abstract, and full-text search as it is a prominent concept for the continuous integration of AI. The term “regulatory” was also added to our search criteria because it is a relevant term for the implementation of AI in health care within the domain of software as a medical device. The reference lists of the included studies were examined to ensure that all relevant papers were included.

The inclusion criteria were articles published from January 1, 2010, to April 13, 2022, that included case studies and guidelines on the implementation of AI analytic tools in hospital settings using EMR data. Given the scarcity of real-world AI tools in hospital settings, especially the scarcity of published case studies of unsuccessful implementations of clinical AI tools, we specifically included case studies that successfully implemented AI within hospitals to understand lessons learned and provide use cases that other jurisdictions may learn from. On the basis of a review of frameworks for AI implementation in health care practice, we defined the term implementation as “an intentional effort designed to change or adapt or uptake interventions into routines” [19]. The term “barrier” was defined as “experiences that impeded, slowed, or made implementations difficult in some way” [20]. In contrast, the term enablers was defined as factors, experiences, or processes that facilitated the implementation process. Studies conducted in community or primary care settings were excluded as our main focus was hospital facilities. Studies that did not use AI models were also excluded. We also eliminated non–English-language and conference articles. Studies that focused on regulatory domains and challenges, opportunities, requirements, and recommendations were also excluded as they did not demonstrate real-world AI implementation. The selection of studies was based on the criteria specified in Textbox 1.

For the screening and data extraction procedures, the Covidence (Veritas Health Innovation) systematic review software was used [27]. A 2-stage screening process was performed with the involvement of 2 reviewers (AKR and OP). In the initial stage, the reviewers assessed the relevance of titles and abstracts based on the inclusion criteria. Subsequently, in the second stage, the full texts of the included articles were reviewed by AKR and OP independently. Consensus was reached through discussion between the reviewers whenever necessary.

AKR and OP conducted the procedure of data extraction. The following study characteristics were extracted from all final included studies: country, clinical setting, study type (case study or guideline), and aim of study. With the adoption of EMR as a prerequisite for AI development, our focus was on extracting evidence of enablers and barriers solely within horizons 2 (implementation) and 3 (creating new models of care). In total, 2 reviewers (AKR and OP) independently extracted evidence regarding enablers and barriers (RQ 1), subsequently reaching consensus through weekly discussions and analysis. The extracted data were disseminated among our research team for review and to gather additional feedback.

To address the second RQ (RQ 2), we mapped the findings from previous reviews along with the found factors in this study across horizons 2 and 3 of the digital transformation framework [15]. Following the data extraction phase, 2 reviewers independently mapped the identified enablers and barriers to 4 categories (people, process, information, and technology). During the mapping of a given enabler or barrier, if it was related to the development of AI analytics, it was mapped to horizon 2 considering its relevance across the 4 domains (people, technology, information, and processes). When an enabler or barrier was associated with the postdevelopment phase focusing on establishing new care models, it was mapped to horizon 3. Consensus was reached between AKR and OP through a meeting to finalize the mapping phase.

For the included use case studies, we used the Mixed Methods Appraisal Tool (MMAT) [28] to conduct a quality assessment. The choice of the MMAT was suitable as the included use case studies exhibited a range of qualitative, quantitative, and mixed methods designs. For evaluating the methodology of guideline studies, we followed the ADAPTE framework [29]. With 9 modules for guideline development, this framework was designed to streamline and enhance the process of creating guidelines within the health domain. The quality assessment was conducted independently by 2 authors (AKR and OP), and any discrepancies were resolved through a meeting.

The search strategy retrieved 1247 papers from PubMed, Scopus, IEEE Xplore, and Web of Science for analysis, and 67 (5.37%) duplicates were identified and eliminated using the EndNote (Clarivate Analytics) citation manager. After screening titles and abstracts, 92.03% (1086/1180) of the studies were removed as the inclusion criteria were not satisfied. A total of 7.97% (94/1180) of the papers remained for full-text review following title and abstract screening. In total, 48% (45/94) of papers were excluded because AI models were not implemented in clinical care. A total of 19% (18/94) of the studies were excluded because they focused on regulatory domains. In total, 9% (8/94) of the studies were excluded due to being the wrong intervention (eg, studies that did not develop AI models). A total of 3% (3/94) of the studies were found to have a clinical population that did not align with our inclusion criteria (eg, hospitalized patients). One study was not in English and was excluded. In addition, 7 studies were discovered by scanning the reference lists of the included articles. In total, 26 studies were included in this review, comprising 9 (35%) guideline studies and 17 (65%) papers with successful implementation examples (Table 2). Figure 2 presents the PRISMA flow diagram outlining the outcomes of this review.

Table 2 outlines the characteristics of the included studies in this review. The publication dates of the included studies ranged from 2019 to 2022 [20,30-54]. In total, 65% (17/26) of the studies were case studies on the implementation of AI in hospitals [20,39-54], whereas the remaining 35% (9/26) were implementation guidelines [30-38].

Of the 26 identified studies, 15 (58%) originated from the United States [20,32,33,36,37,39,43-45,47-49,52-54]; 2 (8%) originated from the United Kingdom [30,42]; 2 (8%) originated from the Netherlands [34,41]; and 1 (4%) originated from China [55], Australia [46], Japan [50], Canada [51], Austria [46], Germany [35], and Sweden [31] each.

Radiology was the clinical setting in 46% (12/26) of the studies [33,34,36,37,39-41,43-45,48,49]. A total of 38% (10/26) of the studies were conducted in general inpatient wards [20,30-32,35,38,46,47,51,52], and 15% (4/26) were conducted in emergency departments [42,50,53,54].

Regarding the 35% (9/26) of guideline studies, none fully adhered to the ADAPTE framework [29]. Although these included guideline studies had clear scopes and purposes aligned with this review, they all lacked details concerning the assessment of quality, external validation, and aftercare planning procedures. The details of this assessment for all the guideline studies can be found in Multimedia Appendix 3 [20,30-54].

With respect to the 65% (17/26) of case studies, they were classified into 3 groups: quantitative descriptive (12/17, 71%) [39,40,42-49,51,54], qualitative (4/17, 24%) [20,41,52,53], and mixed methods (1/17, 6%) [50]. Overall, 5 of the case studies met the MMAT criteria: all 4 (80%) qualitative studies and the one mixed methods study. The remaining 71% (12/17) of quantitative descriptive studies failed to fully adhere to the MMAT criteria. In all but 17% (2/12) of these quantitative descriptive studies, an appropriate data sampling strategy was not used to represent their target population [40,49]. The statistical analysis of the findings was assessed as appropriate in 58% (7/12) of the quantitative descriptive studies [42,43,46,47,49,51,54]. Overall, our assessment revealed that the quality of 81% (21/26) of the included studies was poor due to insufficient reporting of their methodologies (Multimedia Appendix 3).

The health care industry needs to adopt new models of care to respond to the ever-growing demand for health services. Over the last decade, the academic community has shown considerable interest in the application of AI to explore new innovative models of care. Despite the numerous papers published each year exploring the potential of AI in various health domains, only a few studies have been implemented into routine workflows. Investigating the factors that lead to the success or failure of AI in health care could potentially provide actionable insights for the effective implementation of AI in clinical workflows. In this review, we explored the current state of the literature focusing on the implementation of AI in hospitals. Our review of 26 studies revealed several enablers of and barriers to the implementation of AI in digital hospitals. Although our search for studies dated back to 2010, all 26 case studies and guidelines found in our study were published from 2019 onward. This is not surprising considering the significant progress made in AI implementation across many fields in recent years. Given such substantial advancements, implementation science needs to be further developed to accommodate these new AI innovations in health care [19]. This paper can serve as a road map for decision makers, presenting key actionable items to translate AI into hospital settings and leveraging it for potential new models of care.

While this paper extends the findings of previous reviews by examining the factors associated with AI implementation in health care [22-24], a significant aspect found in both previous reviews and our study underscores the significance of process-related factors for creating AI analytics. A large number of papers identified in this study (22/26, 85%) reported process factors as enablers of their AI implementation, aligning with the factors found in all previous reviews (enablers 3-9; Table 3). This commonality indicates the significant opportunity for hospitals to leverage their existing workflows as a strategic approach to enable AI adoption. In the context of developing innovative care models through AI analytics, obstacles associated with people (barriers 14 and 15; Table 4) were identified in 19% (5/26) of the included studies, consistent with findings in 2 previous reviews [22,24]. This highlights the influence of human factors in facilitating the integration of AI in practice.

Apart from the common findings between this and previous reviews, there are several novel aspects to this study. First, it centered specifically on hospitals, the largest and richest source of clinical data. Second, it incorporated AI implementation guidelines from the included studies, allowing for a broader understanding of AI implementation. Third, our review identified new enablers of AI implementation regarding technology and information that can facilitate AI implementation, including quality of data, shared infrastructure for continuous development, and capabilities regarding hardware resources. Fourth, this paper identified new barriers to AI implementation, with most of them being within the domains of process, information, and technology. These barriers included challenges such as data privacy, dealing with heterogeneous data, limitations with the customization of AI analytics, and ambiguity surrounding the design of alert definitions. Finally, the study findings were mapped to a 3-horizon framework encompassing 4 key categories: people, information, process, and technology. This framework offers a clear and practical road map for health care organizations planning to create new AI analytics.

It is important to note that, while our primary focus was on hospital facilities, the findings of this review may exhibit variations across other health care settings. For example, the incorporation of AI in outpatient care may demand different technological infrastructures to enable AI development. Future research can expand upon this study by investigating the evidence of enablers and barriers associated with AI implementation in wider health care settings, including primary care and outpatient care, as we expect that the outcomes of this study may differ in other health care settings. Moreover, the incorporation of studies related to regulatory aspects can be a crucial component for a more comprehensive understanding of the trajectory of AI adoption within health care systems.

Our search strategy identified 26 studies that met the inclusion criteria. All 26 studies were conducted in high-income countries. As a result, the diversity and applicability of the findings to other health care systems were constrained.

By excluding regulatory frameworks from this review in the rapidly evolving regulatory landscape, we may have limited the important implementation guidelines that ensure patient safety and ethical use of AI provided by health care regulatory bodies.

We conducted a thorough examination of the reference lists in the included studies to ensure the inclusion of all relevant papers. Despite a valid research methodology, this approach may introduce publication bias, a factor to consider when appraising the study’s findings.

The methodological reporting of most studies included in this review was assessed as poor, potentially limiting the quality of the findings of this study. While consensus discussions were held after the quality assessment to mitigate potential discrepancies in the final evaluations, it is worth recognizing that this process is subjective and the perspectives of reviewers may evolve over time, resulting in variations when assessed by different individuals.

Although our intention was to identify successful implementations, it is possible that we missed significant enablers or barriers present in failed implementations.

This review incorporated the identified enablers of and barriers to the implementation of AI into a 3-horizon framework to guide future implementations of hospital AI analytics to evolve practice toward an LHS. Successful AI implementation in hospitals requires a shift in conventional resource management to support a new AI implementation and maintenance strategy. Using analytics to enable better decisions in hospitals is critical to enable the ever-increasing need for health care to be met.