There has been an increasing interest in the use of artificial intelligence (AI) systems in health care in recent years as part of supporting health care professionals and individuals with health problems to facilitate care processes. This technology is taking advantage of the abundance of health care data to assess and support decision-making across multiple dimensions. Numerous studies have investigated the prospective utility of AI systems in augmenting decision-making across various health care domains. These encompass early detection, diagnostics, treatment modalities, and self-care initiatives [1-3]. While extensive research has examined the ethics, performance, and potential applications of AI models [4-6], there remains a notable gap in the literature regarding the integrative and empirical use of implemented AI systems in health processes.

AI systems for decision-making are equipped to enhance health care and self-care processes. These systems not only have the potential to improve the efficiency of health care processes but also act as a bridge, integrating health care with self-care by enabling seamless information exchange between the 2 domains [7]. From a health care viewpoint, AI systems hold transformative potential for the future, particularly in the identification and diagnosis of health problems, as it can introduce more objectivity and precision [8]. Self-care, in contrast, refers to the personal management of health and daily life activities outside the formal health care system, while self-help encompasses the use of external resources for the management of health and well-being, both within and beyond the formal health care system [9].

From the standpoint of clinical workflows and end users, AI systems present several advantages, including personalized decision-making, enhanced patient-physician communication, and support for shared decision-making (SDM) [10-12]. However, concerns have been raised by both clinicians and patients regarding the widespread adoption of AI systems, underscoring the need for cultural adjustments within health care to facilitate acceptance [10]. Clinicians have expressed reservations about integrating AI systems into decision-making processes, given the challenges around accepting a second opinion without validation from another clinician [12]. Beyond issues of acceptability, the implementation of AI in SDM poses various challenges, including potential delays in decision-making, communication barriers between health care professionals and patients, and uncertainty surrounding the role of AI in decision-making [13]. Research into the use of AI systems in the context of SDM is still in its nascent stages, emphasizing the need for further exploration into how AI systems can effectively support SDM processes in the future [14].

The potential use of AI for SDM holds particular significance in the context of mental health care given the challenges in the identification and management of mental health problems faced by the contemporary mental health care institutions. However, the complex and heterogeneous nature of mental health problems presents considerable obstacles for AI systems in delivering accurate diagnostics, effective treatment guidance, and tailored interventions. Instead, AI systems currently demonstrate greater efficacy in aiding the identification of symptoms of mental health problems rather than diagnosing specific disorders [10,15]. Given this context, it becomes imperative to understand the interplay between the complexities of mental health problems and AI decision-making support mechanisms, emphasizing the incorporation of the perspectives of end users. Such an inclusive approach is essential for guiding the effective implementation of AI systems in mental health settings [16]. Numerous studies underscore the necessity for a nuanced understanding of AI system use within real-world mental health care to inform future implementation strategies [1,17-19].

Despite the potential promises and inherent challenges associated with AI systems in supporting decision-making processes within mental health care, there remains a notable research gap concerning their practical utility. Fundamental questions persist regarding the types of decisions AI systems can support in real-world environments, the extent to which these systems engage patients in the decision-making process, and the methodologies used for their evaluation. Given this backdrop, this study aims to explore the empirical evidence for the use of AI systems to support different types of decision-making in mental health. This includes how it has been researched, applied, and evaluated for use and implementation, contributing to the understanding and improvement of future design, development, and implementation of AI systems in mental health.

A scoping review approach was considered the preferable method to address the explorative nature of the study and to systematically map the available empirical studies within the study domain. The design of this scoping review was guided by the 5-phase framework put forward by Arksey and O’Malley [20] to ensure quality and a systematic approach to the study. The framework phases included defining the research questions (RQs); identifying relevant studies; selecting the study; charting the data; and collating, summarizing, and reporting results. The reporting process was conducted following the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Extension for Scoping Reviews) checklist [21], as shown in Multimedia Appendix 1.

A multidisciplinary research team comprising experts from health care, nursing, AI systems, and implementation research engaged in discussions in the initial phase of the study to identify the scope of the research and RQs. Four main concepts were defined to guide the study: mental health, AI, decision-making, and implementation (Table 1).

The following three RQs were formulated to address the aim of the study:

A search strategy was developed within the research team with the assistance of a librarian. Literature searches were carried out using the 5 main health research databases: PubMed, Scopus, PsycINFO, Web of Science, and CINAHL. Keywords used in the searches were guided by the following concepts and Medical Subject Headings terms: mental health, artificial intelligence, decision-making, and implementation. Synonyms were joined by the Boolean operator OR; next, we combined the search strings for each keyword with the Boolean operator AND (Multimedia Appendix 2). Eligibility criteria were followed for the search and study selection, as shown in Textbox 1.

All articles identified from the searches were uploaded to EndNote (version 20.1; Clarivate) to remove duplicates. The remaining articles were then uploaded to Rayyan (Rayyan Systems Inc), a web-based tool used to screen articles on the abstract level and identify eligible studies for full-text screening. Rayyan was selected as it provides a systematic approach and facilitates organized teamwork [29]. In the first phase of screening, researchers (HA and MN) reviewed the abstracts and held regular follow-up meetings to ensure criteria consistency. In the second phase, full-text articles were reviewed by 2 reviewers in the research team and screened to meet the eligibility criteria and address the RQs. Throughout both phases, articles where there were uncertainties about meeting the inclusion criteria were discussed among the full research team until a consensus was reached. The screening process followed a decision tree process for excluding the studies that did not meet the inclusion criteria, then illustrated in a PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) diagram.

A template sheet was developed to extract data from each of the included articles regarding the characteristics of research to respond to the first RQs, including the author’s name, publication date, title, aim, country, study design, care settings, mental health problems, and types of AI. A descriptive summary was performed to answer RQ1 concerning the characteristics.

The reporting of the results for RQ2 was performed in 3 parts. First, we categorized the AI systems described in the included articles according to their utility in supporting decision-making in mental health. Second, we mapped the AI system’s utility of supporting decision-making in relation to the end user’s interaction and use. This mapping was carried out in terms of types of interaction and decision-making stages as per a conceptual model [24,25], which describe decision-making as a cyclical process spanning from situational awareness to the execution of decisions. Third, we conducted an explorative analysis, mapping data flow and user interaction in relation to the AI system’s basic structure as elucidated by Sremac et al [30], comprising 3 primary phases: training data, AI model, and AI output (decision). In the identified articles, the study by Sadeh-Sharvit et al [31] did not provide sufficient data in the reporting for RQs 2 and 3.

An abductive analysis was conducted for RQ3 [32] combining inductive and deductive reasoning when looking at the data by firstly conducting inductive thematic analysis and connecting the emerged themes deductively to the human, organization, and technology-fit (HOT-fit) framework [33]. This framework provides 3 essential components when an AI system is being used; the human factor focuses on the individual user experience of the AI system; the organization factor focuses on the structure and environment evaluation of use; and the technology factor focuses on the quality of use of the system and the information provided by the AI system. These factors consist of subcategories or dimensions, and each dimension includes several elements. First, the content in each article was collated by the first author (HA) and inductively abstracted into themes. The framework was then used to deductively map the themes into the factors and their included elements. Two new elements, trustworthiness and explainability, were found in the abductive analysis and were included because they were relevant to the study objectives. Explainability refers to the ability of the AI system to provide clear and understandable reasoning of its output or decision support provided to the intended end users [34]. Trustworthiness refers to the extent to which the AI system is trusted by the end users to use, including being reliable, ethical, and dependable for use and in supporting decision-making [35]. The analysis was iteratively and repeatedly discussed between all the authors until consensus was achieved.

The literature search resulted in 1773 articles between the years 2011 and 2022. Following the removal of duplicates, 1217 articles remained for abstract-level screening. Of these, 1159 (95.23%) articles were excluded based on the inclusion and exclusion criteria, leaving 58 (4.77%) articles for full-text screening. A total of 12 (0.99%) articles were found to meet our study eligibility criteria after the completion of the screening process as seen in the PRISMA flowchart (Figure 1).

The articles included were published between 2020 and 2022, and a variety of designs were used (Table 2). Most of the articles were from the United States (7/12, 58%); 25% (3/12) were from Canada, 8% (1/12) were from the United Kingdom, and 8% (1/12) were from China. The included studies encompass 3 settings of investigation: health care, self-care (remotely), and simulation environment. The health care studies took place in primary care, emergency department, and specialized mental health care. The self-care studies focused on AI systems using mobile apps or wearables with sensors. Simulation studies were conducted in the psychiatry department within a university-based setting. The spectrum of severity and types of mental health problems found to be supported by the AI systems varied. Depression was the most common mental health problem targeted by the AI system support (5/12, 42%), followed by substance misuse (2/12, 17%), autism spectrum disorder (2/12, 17%), suicide and mental health crisis (2/12, 17%), and psychotherapy for different mental health problems (1/12, 8%).

This study explored the empirical evidence surrounding the use and implementation of AI systems for decision-making support in mental health. The identified studies primarily focused on AI systems developed for health care and self-care settings, with articles predominantly sourced from high-income countries. The state of research on AI systems in mental health is in the preimplementation stage in clinical processes. Although studies have evaluated the use of AI systems and their potential impact on clinical processes, none of the studies have explored the full adoption or long-term implementation of AI systems in care settings. Of the 12 studies, 8 (67%) evaluated the use of AI systems in real-time care use, while 4 (33%) studies focused on simulations or clinical case studies.

A total of 3 main types of AI systems were categorized according to their utility in mental health: diagnostics and predictive AI, treatment selection AI, and self-help AI. These systems exhibited differences in their use process and interaction with end users. The findings about the types of AI systems and the variety of end users engaged in using these systems were found to be consistent with previous research [47-49], which underscored the potential use of AI in mental health, and emphasized the importance of understanding the sustainable design, which enables the collaboration in decision-making and the integration of these technologies into clinical processes. The AI systems provided support to a diverse range of actors both in health care, including primary care physicians, community psychiatric nurses, psychiatrists, and occupational therapists, as well as outside the health care services, including patients, individuals with mental health problems, and caregivers.

Evaluating the AI systems through the lens of HOT-fit highlighted several challenges in the integration and use of AI systems in care flow. These challenges included potential issues affecting accuracy and use biases, increased demand, physician-patient communication, and engagement with the AI system in self-care. However, despite the existing insights, there remains a pronounced lack of evidence that can help assess the integrative use and implementation of AI systems in different care contexts.

The studies demonstrated the diverse interaction dynamics between end users of AI decision-making support systems. Interactions were influenced by the types of users and the data flow within the AI systems. There were variations in how data flows through the 3 types of AI systems. From the methods used to input data, to the types of interfaces presented to users, to the inner functionality of the AI models, and finally, to the nature of the output data—each system painted a unique picture. These variations align with the 4 central human-computer interaction (HCI) aspects identified by Rundo et al [50] for investigating decision-support systems, emphasizing the necessity of integrating HCI principles into AI technology development and implementation to ensure they effectively serve user needs, enhance decision-making outcomes, and ensure the sustainable use of AI in decision-making in mental health care.

Diagnostic and predictive AI systems in the articles showed the greatest diversity in data sources, which can complicate user understanding and create a black-box effect due to lack of explainability, which can reduce user acceptance and decision-making quality [51,52]. Furthermore, how the AI model functions to support decision-making based on data input is crucial for acceptance and sustainable use [12,53]. Nevertheless, none of the studies showed how the AI model is being sustainably trained or how the end user’s use of the AI systems contributed to the continual learning process of the AI model. This process is a fundamental part of AI or ML apps that provide continual adaptation of the AI model for improvement of its performance with further use [54]. End user–inclusive strategies, such as the human-in-the-loop approach, where health care professionals and patients actively participate in providing feedback to enhance the AI model’s functionality, can be useful in addressing this knowledge gap. This ensures that both humans and technology are aligned toward common long-term goals, promoting their joint improvements and effectiveness [54,55].

Three main types of interfaces for end users were discernible when examining the use and data flow within the AI systems: data collection interfaces, AI output visualization interfaces, and combined input and output interfaces in which interaction loops take place as in self-help AI type. These interfaces were designed to be used by a single user (patient or health care professional) or multiple end users. The differences in interface types and end-user interaction can pose challenges in examining these systems and ensuring fluency in care flows. For example, while self-help AI interaction loop interfaces can simplify the flow of use and enhance automation, compared with separated input and output interfaces, it can also complicate measuring interaction effectiveness in the health care [56-58]. Moreover, when multiple end users with different backgrounds use a single AI interface for decision-making, there is a growing need for interfaces to provide on-demand customized explanations to fit the understanding and reasoning of multiple stakeholders [59]. This also points to the need for co-design approaches to involve stakeholders from the early stages of development [60-62].

Using AI system interfaces to support decision-making presents varied dynamics of interaction, not only between the end users and the AI system but also among the end users themselves. Interestingly, none of the reviewed studies included an AI system explicitly designed to facilitate SDM between health care professionals and patients nor did they assess the role of SDM when using AI systems. This absence raises questions about the extent to which collaboration between health care professionals and patients, often highlighted as crucial for improving decision quality in mental health care, is being integrated into AI-assisted processes [51,52,63]. Drake et al [64] emphasized the need to incorporate SDM into the clinical workflows for effective implementation of patients’ involvement [64]. As AI systems become more integrated into clinical decision-making workflows, they could alter traditional clinician-patient interactions, introducing new modalities such as clinician-AI, clinician-patient, and patient-AI interactions [65]. Each of these modalities may bring normative challenges that affect the roles of agency, transparency, trustworthiness, and responsibility among end users. Future research should explore how to thoughtfully integrate SDM in the design, implementation, and practical use of AI systems in care processes, while also addressing the broader normative shifts that may arise in clinical workflows because of AI integration.

The evaluation of the 3 types of AI systems described in the articles using the HOT-fit framework highlighted several challenges impacting the use and potential implementation of these systems in mental health care processes. General challenges were observed affecting the accuracy of the AI systems due to the use of nonrepresentative or skewed data input, in addition to the selective use of the AI system participating in potential future biases. Other challenges found more in one AI system than another included the increase in usual care process steps when using the predictive AI systems, the need for adjustment for patient-physician communication when using the treatment selection AI type, and user engagement issues when using the conversational agents in self-help AI.

It was noted in some studies that health care professionals had a selective use of the AI systems for cases with specific severity, or the systems were preferred to be used by specific health care professionals more than others. This selective use with the continuous learning of the AI model may lead to feedback loop biases. This type of bias refers to the distortion of the AI output affected by the skewed use for specific cases over an extended span of time. Consequently, the AI output becomes more accurate to the selected cases and less toward the general use, which can affect the overall accuracy [66]. Furthermore, users in some of the studies described that the AI systems are useful for purposes that deviate from the AI model’s primary objective. If the potential deviated use was not considered during the implementation efforts, this issue can lead to concept drift bias. This refers to the bias that may happen when the AI processing method differs from the actual use of the AI system leading to degradation of the AI output performance over time [67]. This underscores the need for including health care professionals and patients in the development and implementation phases to ensure that the AI system is designed and implemented in alignment with the sustainable use dynamics [68,69]. Co-design methods for analyzing sociotechnical scenarios can facilitate this by engaging stakeholders in early phases using prototype drafts to validate key AI-related concepts such as its goal and potential risks [62]. Early stakeholder involvement is also essential for building trust and ensuring implementation and adoption. Research shows that collaboration among a diverse, interdisciplinary team, including clinicians, potential AI users, hospital leaders, quality improvement teams, human resources, and IT departments is key to achieving effective implementation. This collective expertise ensures that the AI system meets clinical needs, aligns with organizational goals, integrates smoothly into workflows, and addresses technical, operational, and user concerns [70].

In diagnostic and predictive AI, the utility of the AI systems in detecting mental health state and identifying potentially hidden cases can disrupt common clinical processes by increasing demand and requiring additional steps or time to validate the decision support presented by the AI system. Nevertheless, none of the studies evaluated how this disruption of the care process can be adjusted to and integrated into the usual care routines. If not addressed in the future, this can lead to a dilemma of consequences between undertrusting or overtrusting the AI output in decision-making. Undertrusting of the AI output may require additional steps to validate the decisions provided by the AI system leading to additional workload and time consumption or abandoning the AI system use [12,71]. In contrast, the overreliance on AI systems, particularly when they have a high success rate, can lead to automation bias, where decision makers favor AI recommendations over other sources information including their own [72]. This may result in increased dependency on AI, discarding both health care professionals’ and patients’ judgments and preferences. Such a shift could move clinical care away from person-centered care toward the paternalized style of decision-making raising ethical and safety challenges [73,74]. A prominent example is a commercial AI model implemented in the health care system in the United States, which led to a racial bias discriminating access to health care for millions of patients [75]. These concerns highlight the need for caution in scaling and adopting AI for critical decision-making tasks without robust evidence. Initiatives such as the AI Act aim to address these ethical issues by setting requirements for AI development and implementation. Current recommendations emphasize that final decisions in diagnosis and care should remain with human judgment, with AI serving as an assistive tool, a view supported by numerous studies in mental health care [8,76-78].

Treatment selection AI systems were the only type evaluating the patient-physician relationship when using the AI system. Health care professionals used the AI system in their communication with the patients in some of the studies to discuss options and guidelines. However, this integration of AI decision support can disrupt the communication process, introducing sociorelational challenges into the clinical processes that need to be addressed in the design and implementation of the AI to avoid such challenges [13]. The average time willingness to use the AI system was limited to 5 minutes in one of the studies. This time constraint may present a challenge of integrating the AI support into the patient-physician communication. It highlights the need for efficient and explainable AI that can facilitate communication within a short span of time and tailored to the individualized care processes considering both technical and end-user preferences [79].

A main challenge identified in the studies with self-help AI systems based on conversational agents is inconsistency in user engagement, including frequency of use, duration, and completion rates. The self-help AI systems, primarily used for psychotherapy and psychoeducation, face distinct challenges in both self-care and health care settings. Low engagement can reduce the effectiveness of self-help AI systems, particularly because behavioral change and cognitive behavioral therapy (CBT) techniques require long-term engagement to attain effectiveness [80,81]. While conversational agents show great potential in facilitating mental health services [82,83], several studies indicate a lack of clarity about effectiveness and evidence-based design for therapy and mental health assessment [84-88]. This challenge highlights the need for additional research on long-term effectiveness and reliability of conversational agents across various mental health applications. Jabir et al [89] examined how conversational agents are typically evaluated and found that effectiveness assessment usually includes at least 1 clinical outcome and 1 user experience outcome. Given the need for sustained engagement with CBT-based conversational agents and the findings of our study, we recommend that future studies incorporate a third type of outcome measurement to better validate effectiveness. This additional measurement should focus on behavioral change, which is a fundamental aspect of CBT and chronic health conditions [90,91]. Cole-Lewis et al [92] demonstrated that engagement with technology can lead to real-world behavior changes and consequently supporting health outcomes. Including behavioral change as an outcome is important to ensure that the mediated behaviors are successfully targeted not just the clinical outcomes or the user engagement. This is crucial because long-term life changes may mislead the association between self-help AI effectiveness and other positive life-related factors. Using behavior change techniques can serve as validated instruments for assessing health or decision-making behavior change, thus guiding the evaluation of conversational agents [93-97].

Overall, from the findings it is evident that successful integration and implementation of AI systems in existing mental health care workflows requires substantial efforts to create evidence regarding its sustainable use. There is a lack of empirical studies evaluating the long-term implementation of AI in mental health care. Nair et al [70] identified strategies and barriers for successful implementation that lead to sustainable use in 3 essential areas (planning, implementing, and sustaining the use). When integrating AI systems for decision-making support, the evaluation of changes in clinical workflows needs to be considered proactively during the planning phase to avoid unseen organizational or increased demand issues when put in place [68]. This step requires the inclusion of multidisciplinary individuals who will engage with the workflow using simulations or prototypes to enable the detection of any barriers or potential risks before putting the integrated AI systems in place [98,99]. Regarding implementation strategies and ensuring sustainable use, it is important to investigate approaches relevant to the context of mental health care processes addressing resistance of change, staff training, and monitoring of AI performance. This can help create evidence participating in planning for sustainable AI systems. Future studies, including case studies and longitudinal implementation research, are necessary to generate evidence for how to successfully plan, implement, and integrate AI for sustainable use in mental health. These studies can support proposing strategies for integrating AI decision-making utility in mental health practices, with the consideration to, including all relevant stakeholders in the research process.

Some methodological considerations are worth noting. The literature search was conducted using 5 relevant databases, which were selected based on the study’s focus and the research field. However, it is important to acknowledge that using different databases may yield varying results, potentially affecting the comprehensiveness of the review. The keywords used in the search strategy were thoroughly discussed within the research group and with librarian assistance to align with the study’s aims. Nonetheless, the evolving nature of interdisciplinary field, such as health care, implementation science, and AI technology means that the literature may use a broad range of terminology. This variability could impact the identification and inclusion of relevant studies, as different keywords might yield different sets of literature.

Furthermore, the decision to include only AI systems with technology readiness level ≥6 intended to focus the exploration on AI systems that are close to the implementation phase and investigating the AI use in care settings. While this criterion ensured relevance to real-world applications, it also narrowed the scope of the review, excluding studies that focus on the earlier developmental stages of AI models. This limitation means that the review does not address the potential capabilities of emerging AI systems that are still in the research or proof of concept phase but might have profound implications for mental health applications in the future. In addition, the limited number of studies available in implementation phase restricts the ability to draw generalized conclusions about the long-term effectiveness or scalability of these AI systems in mental health.

Moreover, 2 researchers independently conducted the selection and review, enhancing the reliability and reducing potential bias in study inclusion. However, assessing SDM in relation to these AI systems posed a challenge, as most studies did not report on the interaction dynamics between health care professionals, patients, and the AI system in a mutual decision-making context. This lack of detailed reporting on SDM limits the understanding of how AI influences or facilitates collaborative care in mental health.

This study sheds light on the knowledge gaps regarding the empirical evidence of the use and integration of AI systems in mental health decision-making processes, offering insights that can guide the development and design of human-centered AI systems. It underscores the need to consider the utility of AI systems and end-user perspectives in the planning for the development and implementation of AI systems to support decision-making in mental health, in addition to the need for more empirical evidence to validate the factors affecting behavioral sustainable use. Furthermore, this study presents the current state of complexity in which the HCI perspective is fundamental in combination with co-design methodologies to ensure person-centered care when implementing AI systems in care processes.

Future research is advised to explore the integration of support for SDM in the AI system’s design and development, addressing the gap identified in the current literature. In addition, there is a need for conceptual refinement of the current SDM process due to the disruptive effect of AI use. As the care process is evolving from the traditional physician-patient SDM, potentially adding the AI component as a third decision maker in the SDM dynamics. This adjustment is leading to changes in the essential elements of the process, and the 3-talk model, in particular when it comes to the technology-human SDM dynamics [28,100]. Conducting implementation studies are needed for creating robust empirical evidence, and to address potential long-term challenges by identifying barriers and facilitators of AI implementation and sustainable use in mental health care.

The scoping review demonstrated the diverse range of uses of AI in the mental health field, including health care settings, self-care contexts, and hybrid approaches. However, there was a lack of empirical evidence from an implementation perspective on how the AI systems should be integrated into clinical processes. None of the studies discussed how to adjust clinical processes to accommodate aspects such as patient-physician communication and SDM.

The results showed that the utility of AI systems in supporting decision-making in mental health varies and illustrates the complexity of evaluating the utility of decision-making, influenced by the type of actors involved in decisions, the level of decision actionability, and the dynamics of data flow through the AI system. The evaluation of the use of AI systems in care settings through human, organization, and technology lenses revealed several challenges for the implementation of AI systems for sustainable use in care settings. As a consequence, these use- and implementation-related challenges may impact the accuracy of the decisions supported by the AI system, disrupt physician-patient communication, pose trust issues, and present engagement problems impacting the effectiveness and adoption of AI decision support. Future studies are needed to address stakeholder’s needs in AI systems design, evaluate the implementation of AI systems in clinical processes for sustainable use, and assess the integration of SDM in AI systems.