Key Themes
Overview
In this section, we describe the experiences of health care professionals in using AI tools to support clinical decision-making, as reported in the included studies. We focused on the nuances we believe are specific to AI following the process described in the Methods section. We identified seven themes: (1) understanding of AI applications; (2) level of trust and confidence in AI tools; (3) judging the added value of AI; (4) data availability and limitations of AI; (5) time and competing priorities; (6) concern about governance; and (7) collaboration to facilitate the implementation and use of AI tools. We provide detailed exploration of each theme in subsequent sections.
Health Care Professionals’ Understanding of AI Applications
In total, 10 studies reported concerns with health care professionals’ lack of understanding of AI applications. This includes lack of understanding of the outputs of the AI application and the algorithms or the rationale for the outputs by the AI applications due to the lack of transparency relating to non–knowledge-based AI applications [24,26,27,30,32,37,38,40-42]. For example [42],
This lack of transparency is perceived as a major challenge; the AI technology represents as a “black box”, and its users have no power to understand its mechanisms, or modify them to tackle potential problems.
As another example, a study on the use of a machine learning sepsis early warning system reported as follows [37]:
Both RRT [rapid response team] nurses and ED [emergency department] physicians said that they lacked the knowledge and understanding required to assess the validity of the machine learning model...Physicians also lacked knowledge about the model and the predictive nature of the model.
Another study on sepsis risk prediction reported confusion regarding understanding what the alert means [30]. In an attempt to minimize this issue, inclusion of an explanation for the alert firing was introduced in some hospitals. Some felt the explanations were helpful while others found it more confusing [30]:
A lot of people get confused...so say you get 25, when the patient’s really sick and then the number goes to twenty, does that mean the patients getting better? What do all of the subsequent numbers mean? If it goes up to 30, is the patient getting worse?
Concise content with explanations for firing was well-received. A few noted that this was not possible or was more confusing, but most felt inclusion of explanations was helpful.
Participants in a study on the use of AI tools in radiology reported as follows [32]:
Dr V: How does [the AI tool] know that this is a nodule, but this isn’t?
Dr C: They all look identical to me.
Dr K: They expressed frustration in their inability to understand the divergent AI results: What is it telling me to look at? At this tissue? It looks just like the tissue over here, which is perfectly normal...I have no idea what it’s thinking.
However, in a study [26] on an early warning system for sepsis, limited understanding of how the AI system operates was not considered a major barrier to the use of the application.
Level of Trust and Confidence in AI Tools
Findings related to trust in the accuracy and judgments or recommendations of AI tools are mixed [19,21,26-28,30-37,39,41,43]. Some health care professionals believe that the AI tool performs as expected while others do not, and this was reported across various AI tools. For example, in a study [30] on a sepsis risk prediction tool, implementers were unsure whether the AI tools were able to predict sepsis with clinically meaningful specificity when compared with the traditional early warning system using a systemic inflammatory response system, which relies on physiological data (vital sign and laboratory abnormalities). Participants noted feeling disappointed about the predictive potential [30]:
The tool...was supposedly predictive, but we discovered...it wasn’t predictive...it was really telling providers that they’ve met the criteria for severe sepsis which...is not really predictive because they’ve already met it. It wasn’t that you were getting it before it happened so even though they were selling it as a predictive model I’m not so convinced it was predictive.
In another study on the use of AI to support delirium prevention, the following was mentioned [27]:
[S]even users (14.9%) did not believe that the application is a useful support for delirium prevention, and seven did not believe that the application can be used to detect delirium at an early stage.
Referring to the use of an AI tool for lung cancer diagnosis, a study reported the following [32]:
Radiologists were deeply committed to providing judgments with maximum certainty, but they expressed difficulty feeling certain given the opacity they experienced when considering divergent AI results: “I just don’t know of any radiologist who’s not looking closely at the case because they have AI. Because at the end of the day, you’re still responsible. How can you trust the machine that much?”
Dr E
In a study among health care professionals on the use of BoneXpert, some health care professionals reported struggling to accept outputs from the application [41]:
Interestingly, in three hospitals, we found that the referring clinicians did not trust the output of the AI application and redid a manual bone age analysis for every scan. Thus, just like the radiologists, the referring clinicians showed varying levels of acceptance of AI applications.
Divergence in the opinions of health care professionals and AI recommendations creates uncertainty. However, by using “AI interrogation practices,” radiologists are able to use AI in a way that helps them to experience less uncertainty in making final judgments [32]:
On the surface, it may seem that using the AI tools (and experiencing opacity) increased the overall uncertainty these radiologists experienced; however, in fact, using the AI tool resulted in radiologists experiencing less uncertainty making their final judgments. They achieved this by using “AI interrogation practices” or practices that human experts enact to relate their own knowledge claims to the AI knowledge claims.
Other experiences that led to lack of trust in AI tools were the lack of relevant empirical evidence on AI clinical performance and of its impact on health care workflow and quality and the problem for clinicians where AI results do not match their own judgments [32,41]. For example, in a study of the implementation of AI applications in radiology across 7 hospitals in the Netherlands, the authors reported the following [41]:
[T]here is a lack of empirical evidence on the effect of AI applications on the radiological workflow, as well as their added value for clinical radiology practice...measuring clinical and organizational benefits of AI on a microlevel is difficult...publications on the validation of the algorithms are based on laboratory rather than clinical settings.
Societal expectations may also influence confidence in AI tool use. In a study of the use of the Watson tool in China [42], the authors reported that social media and organizations frequently talk about AI and attribute “magic” qualities to AI, which often leads to doctors being disappointed:
Hospital managers or doctors report to experiencing frustration when facing the real technology after the societal hype.
In one study [43], participants suggested that having a track record of the actions of health care professionals based on AI recommendations and the corresponding patient outcomes could facilitate confidence in AI tools.
Judging the Added Value of AI
In some studies, health care professionals reported that AI tools provided added value and improved decision-making [19,20,24,27-30,35,36,39-41]. This included automation of burdensome tasks [41], avoidance of mistakes [32,41], reduction in variability in clinical practice [22], improvement in team communication [24,36], reduction of alert fatigue due to the combination of multiple data points [30], improvement in data collection [20] and team performance [19,27], and provision of more objective and precise diagnosis or prediction [41].
In some studies, some health care professionals found that AI tools served as a reminder or confirmation of their clinical judgments [23,26,27,29,32,35,38,39,42]. For example [27],
The prediction helps to corroborate my own estimation when seeing a patient.
However, not all users of the same tool found it useful [27]:
Opinions about the application’s usefulness for their own work were mixed: 17 users (36.2%) reported the application to be useful for their work, while 15 users (31.9%) did not find it useful.
Health care professionals reported some problems relating to AI technologies not addressing issues that are deemed important to them. For example, health care professionals thought postoperative nausea and vomiting (PONV) has low burden and should not be excessively treated [31]. In a study in a US hospital among health care professionals using AI tools for diagnosing breast cancer, lung cancer, and bone age determinations [32], radiologists said AI tools identify abnormalities that they might not see but which were not clinically significant to impact their final judgment:
Calcifications can be really little and sometimes hard to see. It [Mammo AI tool] sees those calcifications better than I do. But it also sees all kinds of calcifications that are neither here nor there.
Dr B
In a study of peripheral arterial disease (PAD) identification algorithm [43], clinicians emphasized the importance of ensuring that the AI tools focus on supporting clinical decision-making by performing additional analysis that the clinicians do not normally do:
I mean, it [the algorithm] has to solve a real problem. Like, I’m not interested in models that in a clinical sense, identify data that I could just identify in the course of my daily work.
Some studies also reported that health care professionals expressed uncertainty about next steps and that AI tools provided no actionable outputs, therefore not adding value to their clinical decision-making [31,38,43]; adding treatment recommendations would have been more useful [31]. For example, referring to a tool to predict the risks of PONV, a physician said the following [31]:
The intuitive use of the predicted PONV [postoperative nausea and vomiting] risk and a stated preference for an actionable recommendation by intervention group interviewees suggested that being presented only with a predicted risk may be difficult to use in a clinical decision. Adding a risk-corresponding treatment recommendation may assist physicians in interpreting the predicted risk for a decision on PONV prophylaxis.
Theme 4G, quote 43
Health care professionals also voiced concerns over AI tools not providing actionable information [38]:
It would have been nice to have some sort of actionable items, because while the information is good to know, I was never really sure what to do with it. It’s like, great, my patient has an elevated one year mortality risk. What can I do about that? What do I do with this information?
RES02
From another study [40]:
They gave us the risk scores, but we did not really understand what to do with this information, what do these scores mean? It took us a while to figure out what is our threshold for what are we going to consider high risk patients.
Readmission Risk Tool user
Furthermore, in a study of an automated mortality prediction model for identification of patients for palliative care, AI helped reduce uncertainties, especially in less-experienced health care professionals [38]. In some situations, AI tools reportedly caused added work for health care professionals [21,29,34,37,41]. For example, in a study on the use of AI for the detection of diabetic retinopathy [21], the AI tool required high quality data and or images, which may be difficult to achieve in some cases. A study participant reported the following:
It gives guaranteed results, but it has some limitations. Some images are blurry, and I can still read it, but [the system] can’t. P3 shared the same sentiment, “It’s good but I think it’s not as accurate. If [the eye] is a little obscured, it can’t grade it.” The system’s high standards for image quality is at odds with the consistency and quality of images that the nurses were routinely capturing under the constraints of the clinic, and this mismatch caused frustration and added work.
In one of the studies on the implementation of AI in diagnostic radiology [41], the clinical benefits or organizational goals for using AI applications were not clearly established before implementation, making it hard to assess after implementation:
From an organizational perspective, clinical benefits or organizational goals that might be achieved by using AI applications are not clearly established ex-ante and therefore hard to assess after implementation.
Health care professionals were concerned that patients may not be willing to have AI tools used for them [42]:
They [the patients] have no idea about Watson [a system for designing personalized treatment for cancer patients]. They will think: why do I need a machine to look at [my problem]? I prefer an expert doctor.
Health care professionals expressed concerns about limiting treatment for palliative care patients based on the risk prediction by AI tools [38]. There were also concerns about clinicians and regulatory agencies becoming overly reliant on AI [26,29]. For example, health care professionals may not actively consider their cases and they may not be able to refine their skills and maintain cultural competence [29].
Data Availability and Limitations of AI
A study reported the users’ concern about the potential for AI to worsen inequity as disadvantaged populations are not represented in health datasets [43]:
About 20% of our population does not have data.... So we will be over-selecting and over-serving 80% and under-selecting and under-serving 20% of our population because of a structural dynamic here in the United States.
And I should also say those even worse, those 20% [who we don’t have data on], or a higher proportion of them are older, coming from an economically higher social deprivation index, higher disease burden, nonwhite.
Concerns about the limits of AI in considering all the aspects of patients’ lives were also talked about [31,43]. For example, in the study [31] on an AI tool for PONV, the following was discussed:
[T]hey [physicians] felt that a prediction model does not take into account all aspects of a specific patient. The prediction model only predicted the risk for a specific outcome and did not weigh the benefit of treatment against the expected harms and contraindications for a particular patient with specific characteristics and comorbidity.
In the study of the use of PAD identification algorithm [43], health care professionals reported having access to more data about patient than AI algorithm uses. There were concerns about the AI tool not being able to account for factors such as patient’s culture [29] and ground context and therefore limiting the potential impact of the AI [43]:
Exercise, nutrition, smoking, and medication adherence are critical components of PAD [peripheral arterial disease] management and clinicians expressed concern about adopting the algorithm to make intervention recommendations without knowing more about these factors...The algorithm’s potential impact was limited by an inability to account for barriers patients faced in managing PAD.
Time and Competing Priorities
Health care professionals reported lack of time to fully utilize AI tools. For example, in a study on the use of the modified Hospitalised-patient One-year Mortality Risk tool, physicians reported lacking time to address the alert because they were often focused on the acute needs of the patient [38]:
The inpatient stay often is very compressed. They’re in the hospital, they’re getting treated, and then they’re home. And so, there is not time during the inpatient stay to address these things.
In the study of the use of AI for PAD identification [43], health care professionals reported limited time to review high-risk patients and having difficulties in choosing which patients to prioritize:
The PAD [peripheral arterial disease] ML-driven CDS can be run on all patients in a population, but only a small number of high-risk patients could be reviewed each week. Stakeholders described the difficult trade-offs they had to make when considering how to pick patients to prioritize for Population Rounding.
Time constraint could be further heightened where AI recommendations differ from the clinical judgment of the health care professionals [32]. However, in a study on the use of AI in diagnostic radiology, radiologists were willing to invest additional time to relate their own knowledge to the AI knowledge claims to build an understanding of the AI results and reconcile divergent views [32]:
I know my limitations and I know this [CT AI] is going to help them [nodules] stand out a little better. It’s worth the extra time in my mind.
Concern About Governance
Health care professionals expressed concerns over regulatory and legal uncertainties surrounding AI use [41,42]. For example, uncertainties regarding legal responsibilities for misdiagnosis stemming from AI recommendations were reported in a study in the Netherlands [41], and this may vary for different countries. In another study [42], a health care professionals reported that it is illegal for AI applications to make clinical decisions in China:
In China it is illegal for an AI system to make a decision.
1HP05
In our hospital, we use Watson to assist the Multidisciplinary Team (MDT).... We discussed how to use Watson for a long time.... As the first hospital to use Watson, we find this way [to use Watson together with the MDT].... But this really gives us a heavy burden! Because when we use Watson, we must have at least five doctors to work together with Watson [as required by regulation]. They [the 5 doctors] will sign on the report.
1HP05
Collaboration to Facilitate the Implementation and Use of AI
In a study [40] on the use of low bed tool and readmission risk tool, the authors described how health care organizations can enable collaboration between key stakeholders (users, developers, and outside experts) to facilitate the development and implementation of AI. This collaboration allowed users (care management and utilization management team members) to be involved in the identification of initial needs, identification of new users, need for experts, identification of potential sources of inaccuracy, and possible areas of improvement [40].
Having health care professionals serve as champions or leads for AI use and having knowledge exchange platforms supported the use of AI [26,41,43]. For example, the study [41] on the implementation of AI applications in clinical radiology reported that local champions (radiologists with keen interest in AI and good understanding of AI applications) are vital in encouraging implementation within their departments. In another study [43], having a physician lead the integration of AI tool for PAD identification was reported to be beneficial in facilitating the translation to create real-world clinical impact:
Dr. XXX obviously was very aware, she was kind of the one that spearheaded [the integration] and wanted to get this going. So I think you know, [targeting clinical need] is her role. I’m sure with her specialty, she probably realized that it was an area that needed more attention.
Operational stakeholder
Furthermore, in the study of the use of AI for early warning for sepsis [37], a new role of a sepsis watch nurse was created. The sepsis watch nurses were the primary users of the tool and were extensively trained in the workflow and the tool. They were responsible for communicating the outputs from the AI tool to physicians to facilitate the translation of the recommendations to patient’s bedside. With regard to the skills required to fulfill such roles, the nurses recommended the following [37]:
When asked about the skills and knowledge needed to be a good Sepsis Watch nurse, the RRT nurses mentioned good clinical judgment, knowledge of sepsis, and critical care experience...RRT nurses also explained the importance of strong communication skills to confidently speak with attending physicians whom they may not personally know...RRT nurses thought that strong computer skills were not necessary for the role, given the simplicity of the app. RRT nurses also recommended recruiting nurses interested in the role and the need to create buy-in through continuous feedback.
In another study [26] on the early warning system for sepsis, the deployment team attended staff meetings and also met with individual users, as requested, to explain the system and provide guidance on using the interface. Users were also able to ask questions about the system’s behavior using a feedback button [26].
In a study [41] on the implementation of AI tools in diagnostic radiology, it was reported that only 1 of 7 hospitals had a formalized innovation strategy regarding AI, although 3 hospitals had a designated innovation manager. It is worth noting that 4 more hospitals were reported to be developing a structured approach at the time of the study. The lack of structured implementation processes lead to substantial variations in how the application is used in different departments [41]:
From a workflow perspective, implementation plans do not specify how the AI application should be integrated into the clinical workflow, which leads to significant variations in the way the application is used in different departments. Furthermore, in all cases, the work done to monitor existing practices or the impact of the implementation of novel technologies on the level of the hospital is currently limited.