Prevention of suicide is a global health priority [1]. Approximately 800,000 individuals die by suicide yearly, and for every suicide death, there are another 20 estimated suicide attempts [1]. Despite the largely preventable nature of suicide, factors such as limited service capacity, variable service quality, and barriers to service access significantly impact the progress being made toward reducing suicide rates [2]. Recent advances in transformer-based artificial intelligence (AI)—the technology that has accelerated the development of powerful large language models (LLMs) that human clinicians and consumers can converse with—have been suggested as a potential solution to enhancing the scalability, accessibility, and personalization of health care interventions [3]. In the context of suicide research and care, LLMs can generate novel insights into suicide risk by parsing language, classifying or scoring text, and comprehending and generating human-like language, with the potential to enhance treatments by improving user engagement with digital interventions. Such improvements can also expand the capacity of crisis support services by providing real-time crisis clinician copilots, automated risk assessments, or expedited triage, enabling the optimized use of human resources. In addition, LLMs can improve the quality of training programs for those who may intervene by mentoring trainees through clinical scenarios and flexibly adopting a variety of crisis-seeking personas for clinical role-play scenarios. Hence, LLM technology might be a critical catalyst for advancement in the field. However, the relatively unexplored nature of the field means that we are only just beginning to understand how LLMs can be harnessed safely and effectively for suicide prevention. In parallel, there is a pressing need to explore how emerging trends in LLM-based research, such as the nature of training data, model types, the contexts of application, and methodological norms from relevant research fields, may shape the trajectory and direction of suicide prevention research (either positively or negatively), now and into the future.

AI refers broadly to machine and computational processes used to execute tasks usually thought of as requiring human intelligence. Natural language processing, a language-focused subfield of AI, has advanced significantly with the advent of transformer-based LLMs [4]. LLMs are the current state-of-the-art technology in computational linguistics, comprising attention-based language encoders trained using massive text datasets, generally with billions of parameters [5]. In addition to language comprehension, LLMs have recently shown their utility in generative language applications, such as generative AI interfaces (chatbots) [6]. The availability of LLMs, such as OpenAI’s generative pretrained transformer (GPT) models, Google’s Bard, and Meta’s Llama, has created unprecedented opportunities for language generation and analysis at scale [7], with LLM applications already becoming widespread across fields such as business analytics, commerce, administration, and education [8].

Under the right conditions, LLMs can demonstrate contextual understanding and content generation that closely mimics human interaction. Accordingly, research into LLM applications in health care settings, where human interaction forms the basis of much of service delivery, is accelerating at a rapid pace; for example, LLMs have been investigated as a potential aid in preconsultation, diagnosis, and the management of disease (eg, infectious diseases and cancer [9-11]), as well as in recommending specialist appointments via SMS text messaging–based self-assessment tools for remote populations [12] and generating patient education materials [13].

Given that language is the primary basis upon which symptoms are reported and assessed in mental health, LLMs represent a significant technological advancement in mental health research. LLMs are currently under investigation for their utility in cognitive behavioral therapy facilitation [14], for emotion identification during psychotherapy sessions [15], and for the detection of positive therapeutic behaviors during motivational interviewing [16]. LLMs have also shown potential benefits in helping individuals understand personal coping styles and in facilitating stress reappraisal [7]. The application of LLMs in the field of suicide prevention is particularly promising, given that traditional methodologies, which have struggled to provide actionable insights into complex constructs such as suicide risk, are complemented by the analytic and generative capabilities of LLMs [17].

In evaluating the promising aspects of LLMs, it is also important to consider that these models present a number of challenges to researchers and end users alike. LLMs are often criticized for the inaccuracy of generated information, reducing trust and credibility [5], a failing that is compounded by the lack of interpretability of LLM functioning resulting from their “black box” architecture. Challenges also exist around data security and privacy; for example, LLMs have been demonstrated to identify individuals from deidentified digital data (eg, location data, medical billing information, or a collection of social media posts and metadata) [17]. When considering application contexts, the generalizability of LLMs is restricted to the degree of representativeness of the training data with respect to age, ethnicity, or even those who use social media versus those who do not [18]. Nevertheless, LLMs hold promise for improving models of suicide risk (eg, via digital phenotyping) [17]; for creating synthetic data [19] to increase sample sizes and statistical power for research on low base-rate events such as suicide; for powering conversational agents that support early triage, crisis support, and help seeking; and for clinician assistance tools, such as automated patient assessment systems, and simulation of crisis scenarios to improve training outcomes. Despite these potentialities, the integration of LLMs into suicide prevention currently trails other fields. Critically, significant gaps must be addressed for the field to move forward in a manner that is safe and acceptable, including understanding the interpretability of LLM outputs, addressing biases within training data, and ensuring ethical deployment within prospective crisis settings.

To date, there have been no reviews examining the integration of LLMs into research on suicide prevention and self-harm. Prior reviews have focused predominantly on research undertaken in adjacent fields, such as applications of machine learning or chatbots to mental health more broadly [20-23]. Although several reviews have explored the use of machine learning [24-27] or AI-based strategies [28-31] in suicide prevention contexts, these have not focused on applications of LLMs specifically. Furthermore, although 1 commentary [32] and a review [25] focused on the integration of computational linguistics or natural language processing more broadly in suicide prevention, these articles did not focus on LLMs specifically, nor have any explored applications of LLMs to self-harm.

The aim of this scoping review was to provide an understanding of current applications of LLMs in the field of suicide prevention, including applications to self-harm. The research questions guiding this review were as follows:

Specifically, we aimed to identify emerging trends in the literature related to the types of training data, models used, and intended objectives.

Our secondary aim was to explore reported potential clinical applications of LLMs (eg, as clinical copilots or to augment suicide prevention training programs) and identify ethical considerations that are crucial as the field progresses. Reported potential clinical applications were sought to better understand the future trajectory of practical LLM use in the field, with ethical considerations identified to understand the important considerations for practical deployment.

A scoping review was selected to map the breadth of evidence across the varied fields from which LLM-based research originates to identify key characteristics of studies, reveal knowledge gaps, and inform future research [33]. In doing so, we aim to highlight emergent trends in the research and address critical clinical and ethical considerations to ensure safe, effective, and equitable research outcomes in this key field.

To meet the eligibility criteria, studies needed to describe the application of an LLM to the area of suicide or self-harm. Although various natural language processing models using machine learning methods exist (eg, support vector machines, Bayesian networks, and random forest algorithms), for this review, we focused on contemporary LLMs, defined as computer-engineered language models that use transformer-based neural network architecture [4]. Although LLM training parameters generally exceed 10 billion, there is no formal consensus on parameter scale [6]. In this review, smaller models (<10 billion parameters) such as early incarnations of the Bidirectional Encoder Representations from Transformers (BERT) and Text-to-Text Transfer Transformer (T5) were eligible for inclusion as they represent language models capable of contextual understanding. Studies describing the use of AI, machine learning, or big data approaches in the absence of LLM methodology were ineligible.

The domain of suicide prevention included, but was not restricted to, areas such as ideation, planning, attempts, prediction, intervention, support, means restriction, gatekeeper training, and public awareness campaigns, as well as self-harm. Studies focusing on LLM applications to mental health or psychiatry that did not specifically mention suicide or self-harm were ineligible. Studies describing quantitative, qualitative, or mixed methods designs were eligible. All studies involving human participants were required to report ethics approval. Source documents were peer-reviewed journal articles or conference proceedings. Conference proceedings were included as they have become the dominant form of published research in computer science (encompassing LLMs) in recent years [38]. Conference proceedings were required to have a comprehensive methodology and results sufficient for replication. Conference abstracts were excluded, as were other reviews and meta-analyses.

Studies using electronic health records (EHRs) were not included in this review. EHRs represent a specific type of corpus representative of individuals who are in contact with the health system. Given that a significant proportion of individuals experiencing suicidal ideation are not in contact with formal health services [39], models that draw primarily on EHR data may generate insights that do not generalize to the broader population of individuals who experience suicidality or self-harm. In addition, substantial work has focused on the use of EHRs for the prediction of suicide risk in recent years [20], with the potential for these studies to populate a stand-alone systematic review. This review sought to elicit an understanding of novel deployments of LLMs, particularly in the individual use context, and to propose future use possibilities or potentialities for support, treatment, or prevention. Studies not published in English were also excluded from this review. There was no limitation on country of origin or publication date; however, the advent of transformer architecture in 2017 naturally limited publications from prior years.

Searches were conducted in 4 databases: PsycINFO, Embase, PubMed, and IEEE Xplore. An initial search was conducted on December 6, 2023. Following abstract and full-text screening, the search was updated on February 16, 2024, to ensure recency of the search prior to final text extraction. Additional studies identified via citation chaining were also included at this point.

The search strategy used index terms and free-text terms to cover two core themes: (1) LLMs and (2) suicide or self-harm. Individual database search strings are provided in Table S1 in Multimedia Appendix 2. Search results were imported into Covidence review management software (Veritas Health Innovation Ltd) [40], which was used for abstract and full-text screening as well as data extraction.

Reflecting prior systematic research [41], title and abstract screening were conducted by 1 author (GH), with a sample of studies reviewed by multiple authors during screening and extraction. In line with recommendations for ensuring reliability and bias mitigation [33], 20.2% (77/381) of the studies identified in the initial search were randomly selected and reviewed by a second author (BT). Agreement was achieved for 71 (92%) of the 77 studies. Conflicts were discussed and resolved with input from a third author (AW).

A similar process was adopted for full-text screening, which was conducted by 1 author (GH), with 20.3% (38/187) of the full texts randomly selected for review by a second author (BT). There was 92% (35/38) agreement between the authors. Conflicts were discussed and resolved with input from a third author (AW).

The data extraction template was piloted with 9% (4/43) of the included studies by 2 authors (GH and BT) in line with prior research [41,42], resulting in >90% agreement. Minor refinements were made to the template to assist later synthesis. Data extraction from the 43 included articles was performed by 1 author (GH), with 19% (8/43) of the studies randomly selected and data independently extracted by a second author (BT). The comparison of extracted data showed >90% agreement. The data extraction template included items under the headings of study characteristics, methods, outcomes, and reproducibility and is available in Table S2 in Multimedia Appendix 2.

Studies were classified across deductively derived categories, with categories based on previous research [42] and with scope for adjustments during the review process if required (eg, the aggregation of specific data sources to report on social media more generally as a source of data). Descriptive statistics were computed and presented in textual, tabular, or graphical format, as appropriate. A narrative synthesis of the included studies was conducted to answer the key research questions.

The search results are presented in the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart (Figure 1). The search yielded 533 studies; after removing 127 (23.8%) duplicates, 406 (76.2%) studies were retained for title and abstract screening. Of these 406 studies, 194 (47.8%) were excluded, leaving 212 (52.2%) for full-text review. After full-text screening, 176 (83%) of the 212 studies were excluded (the reasons for exclusion are shown in Figure 1), and 36 (17%) studies met the inclusion criteria. An additional 7 studies were identified through citation chaining, resulting in 43 studies included for synthesis.

A full list of included publications and associated results data is presented in Multimedia Appendix 3 [43-85]. The number of publications by year demonstrated a compounding pattern, with the first publications emerging in 2019 after the development of the transformer architecture [4] in 2017 and then increasing sharply in 2023 after the release of ChatGPT to the public in November 2022.

The articles identified for this review came from 20 different sources that predominantly came from 2 fields (determined by reviewing individual journal aims and scope): computer engineering (23/43, 53%) and health (18/43, 42%). The bifurcated nature of the publication field of origin demonstrates the cross-disciplinary nature of LLM research. Approximately a third of the studies (16/43, 37%) were published as conference proceedings via IEEE Xplore, reflecting publication norms in computer engineering [38]. The largest proportion of studies came from the United States (10/43, 23%). Study funding was predominantly sourced through grants (23/43, 53%). Of the 43 studies, 4 (9%) indicated no funding, and 16 (37%) did not provide funding information.

The studies included in this analysis exhibited a notable divergence in disciplinary focus, with approximately equal representation from computer engineering (23/43, 53%) and health-related (18/43, 42%) fields. The bifurcation of domain expertise in this emerging field has important implications for the safety, effectiveness, and real-world impact of the research outputs, particularly considering the differences in technical training, publication norms, and approach to validation and evaluation used across the different fields; for example, only recently has there been a call to integrate education in the technicalities and ethics of AI into mental health training programs, and many mental health researchers may not yet receive technical training in AI to be able to develop and apply LLMs for clinical research purposes. In parallel, while computer science researchers have significant technical training to enable the development of innovative AI models and access to industry partnerships that can support the scalability of new AI-based tools, the real-world usefulness of these innovations hinges on a deep understanding of the ethical and clinical context in which they are to be applied, the facilitators and barriers to their use, and the needs and priorities of end users—areas of expertise where clinical training is often paramount. Recognizing and addressing the complementary strengths and gaps in these divergent disciplines will be crucial for ensuring that LLM innovations in suicide prevention are grounded in technical excellence and clinical acumen.

Similarly, the dissemination of research from computer science often outpaces the dissemination of research from the field of mental health owing to the stronger focus on conference proceedings with expeditious publication timelines. The accelerated dissemination of LLM-based suicide prevention research originating from computer science fields may mean that practices focused on innovation—a crucial benchmark of research impact in computer science (eg, via patents) [87]—come to dominate early research advancements in this area [88]. Although innovation is crucial for achieving technological advancements, this innovation often comes at the expense of comprehensive clinical validation [89], and there has been growing emphasis on the need for frameworks for validating new AI tools in health research [90].

Recognizing and navigating these disciplinary disparities is essential. Encouraging multidisciplinary collaboration among experts from diverse backgrounds is likely to be critical for bolstering the quality, safety, and impact of research on LLMs for suicide prevention. This could be done by fostering the development of agile methods for disseminating validation research (such as through cross-disciplinary repositories and benchmarking datasets), establishing sound ethical and safety frameworks that cut across key disciplines (such as living guidelines), and facilitating ongoing education and cross-disciplinary training for researchers across both fields. Crucially, prioritizing and supporting studies led by multidisciplinary teams that rigorously assess safety and effectiveness is imperative for ensuring that research efforts lead to broader societal benefits.

There was minimal variation in the base LLM model applied in the identified studies. Most of the studies (35/43, 81%) used either the base BERT model or some variation of a trained or fine-tuned BERT model [91]. The relatively early release date (2018), open-source availability, compact size of this model compared to subsequent models such as GPT, and its adaptability through fine-tuning for diverse applications are all factors assisting the widespread use of BERT in this research area. While open access is critical for facilitating replication and democratizing access to this modern technology, an important consideration is that open-access LLMs trained on generalized datasets might inadvertently perpetuate unwanted biases; for example, BERT has been shown to exhibit stereotypical biases in areas such as gender, profession, race, and religion, with this bias not related to the size of pretraining corpora but likely due to the nature of the training data [92]. In the same study, GPT-2 was found to have less bias, hypothesized in part to be due to antistereotypical associations present in the training data. LLMs are “stochastic parrots” [93] in their generative responses, their outcomes dependent on the quality of the input training data. Therefore, the testing, application, and comparison of multiple models is important to detect and mitigate model biases permeating an entire discipline. Promoting the use of a diverse range of models to assess performance variability [58,83] and uncover potential biases [92] could enhance the strength of research outcomes in suicide prevention. In addition, there is a pressing need to develop specialized open-access models (such as those being developed in medicine) [94,95] for mental health and suicide prevention. These models should draw upon the expertise of mental health professionals, cross-disciplinary researchers, ethicists, and individuals with lived experience. Establishing standards and policies that ensure the transparent reporting of model training data and inherent potential biases is also imperative to enable the critical evaluation and validation of model suitability [7], particularly for self-harm and suicide prevention contexts.

Most of the studies (31/43, 72%) used data from social media platforms such as X (formerly Twitter) and Reddit to train LLMs. Although these are rich data sources, a reliance on social media data as a form of training data may lead to limitations in the usefulness of LLMs. To protect their users, social media platforms often implement policies to detect, suppress, or remove content related to suicide or self-harm. Therefore, using social media data for LLM training may result in an underrepresentation of critical data that would enable LLMs to effectively recognize and respond to expressions of suicidal distress. In addition, algorithms embedded within social media platforms are designed to promote engaging content. As a result, these algorithms often promote user posts that are biased toward certain viewpoints and controversial or sensational topics, while simultaneously minimizing diversity of perspective. Accordingly, the algorithms in social media platforms may drive artificial patterns in the nature and frequency of certain types of user post data that do not reflect real-world interpersonal interactions.

It is also crucial to evaluate whether the profiles and behavior of individuals who post on social media platforms align well with those of the potential end users of LLM-based suicide prevention interventions; for example, while Reddit’s user anonymity may constitute a perceived benefit to users, evidence indicates that users engage in differing behavior when posting anonymously compared to when their identity is known [96]. To mitigate this, it is imperative to curate training datasets that are diverse and balanced [97], as well as representative of the population that the LLM is intended to serve [7]. The inclusion of input from those with lived experience is important during the development of these datasets because these experiences can provide valid human-generated data for model training. Furthermore, lived experience can provide a valuable perspective in reviewing LLM-generated data, enabling key human-in-the-loop validation of otherwise unavailable data for model training (eg, crisis care transcripts). In addition, actively seeking out content, including positive interactions, supportive conversations, and safe discussions around self-harm and suicide prevention, may help enhance the effectiveness of LLM-based suicide prevention interventions.

The studies included in this review predominantly deployed LLMs for the identification, detection, classification, and prediction of suicide or self-harm risk. These areas represent a narrow band of potential use, and despite research focused in this area, there remains a lack of solutions that implement these models in clinical settings [55]. One reason for this is that computer scientists, who are at the forefront of technological innovation in this area, do not generally have in-depth knowledge of clinical processes and workflows used in the mental health field. This limits their ability to envision the broader applicability of LLMs in mental health settings and results in a tendency to gravitate toward more familiar applications, such as classification and regression. Similarly, suicide prevention researchers often do not possess the technical expertise required to adapt LLMs for their specific needs. Consequently, multidisciplinary collaborations are critical to the development of sound research programs that combine the clinical and technical expertise necessary for high-caliber research. As this research advances and models continue to improve, those focused on clinical areas may be deployed in various contexts. However, it has been noted that this may be insufficient and may not reduce suicide attempts or deaths unless the treatment needs of people identified as being at risk are met [98]. Hence, it is imperative to explore innovative solutions in clinical care, consultation, and therapy.

This review highlighted a small number of recent applications that move beyond identification and prediction by leveraging the generative features of LLMs. These applications included assessing whether LLMs could provide access to suicide-related educational information [46], whether they could detect escalating suicide risk and the need for human intervention [61], and whether they could assist in generating mental health nurse care plans for individuals who self-harm [81]. These studies showed that, at present, LLMs can perform well on specific elements of these roles, including providing immediate access to accurate information relevant to suicide prevention [46], using health theory frameworks to construct detailed care plans with tangible goals for clients [81], and generating training aids for mental health providers [81]. However, these studies also highlighted areas where LLMs currently fall short. Specifically, they showed that LLMs did not effectively detect and respond to signs of escalating clinical risk by referring into care [61]. Moreover, concerns were noted regarding how the accountability of health care providers may be impacted by the use of AI-generated mental health care plans [81]. These examples highlight the critical role of human expertise in the deployment of LLMs in suicide prevention contexts, and an important area for future research will be to determine optimal approaches to implementing human-AI collaboration in suicide prevention settings, including through the use of copilot or human-in-the-loop systems that are beginning to be explored in other areas [89,99].

The secondary aim of this scoping review was to consider the clinical applications and ethical considerations of LLMs in suicide prevention practice. Among the studies describing clinical applications of LLMs, “enhanced detection of suicidality” and use as a “clinical assistance tool” were the most commonly presented clinical applications. The ability to identify and predict suicide has remained at near-chance levels for many years [24], and improvement upon this with the help of novel technologies such as LLMs can provide a valuable contribution to the field. However, accurate risk detection alone is insufficient and must be paired with effective and scalable interventions [24]. The use of LLMs as clinical assistance tools can support clinical professionals by aiding with a myriad of tasks [48,57,66,70]. The conception and design of these applications should start with the end point in mind to best plan for the translation of research results into implementation in a clinical setting. Critically, researchers should engage clinicians, health professionals, lived experience representatives, and other relevant stakeholders early in the research project development to ensure that the research can meet the needs of end users. In the case of lived experience representatives, it is imperative to understand how they, as representatives of end users, interact with the various potential applications of LLMs to ensure safety and effectiveness. Clinical applications described in the studies include assisting clinical decision-making by providing a second opinion [66], functioning as a clinician copilot [48], serving as a mental health triage tool [57], and supporting therapy delivery [70]. However, these applications were limited to hypothesized future uses [48,57,66] or pilot studies [70]. More investment is required to develop implementable clinical assistance tools, particularly in supporting the integration of multidisciplinary teams to collaborate on areas such as clinical training and support as well as identifying novel use cases with the potential to positively impact suicide prevention practice. Importantly, LLMs should not replace clinical judgment but rather supplement it, aiding clinicians and health professionals to make more informed decisions [66] by enhancing access to information (eg, suggesting differential diagnoses, proposing alternate hypotheses, or collating scattered information into more coherent forms to facilitate human interpretation). LLM applications should not exceed this human-in-the-loop support-based role until their safety and effectiveness can be clearly demonstrated.

It is also important to consider the context in which an LLM-based application will be implemented, acknowledging that its success may be limited by existing clinical workflows [100] and the overall capacity of the health system to integrate new tools and provide relevant training [101]. As with all new technological advancements in care, LLM integration into health care practices requires strategic integration into existing—often outdated and practically constrained—care models if we are to realize their full benefit. Forward thinking and responsive policy is required, led by governments that engage with stakeholders to map policy and standards to provide for the harmonious integration of LLM clinical support tools, given the accelerating use and potential of AI.

Research indicates that a majority of individuals do not engage with formal mental health services before suicide, potentially due to stigma or fear of judgment [39]. While none of the identified studies deployed LLMs to directly support or intervene with individuals in distress, the ease of access that a language-based user interface offers may help some overcome stigma-related barriers to engaging with formal mental health services. Indeed, preliminary evidence for the acceptability of generative LLMs in support contexts has been demonstrated; for example, evidence shows that a large majority of individuals (78%) would be willing to use ChatGPT for self-diagnosis or to aid in self-managing their symptoms [102]. Some individuals also indicate a willingness to disclose information to nonhuman agents citing reduced fear, less need for impression management, and greater ease in expressing the severity of their emotions [103].

This review found evidence that LLMs can deliver accurate information relevant to suicide prevention [46] and support clinical needs in related contexts such as mental health evaluations, therapeutic consultations, and patient education [104]. Therefore, LLMs hold the potential to offer individuals quick, empathic responses to mental health queries [105], while avoiding potential or perceived judgment and stigma, at little to no cost, available 24/7. Nevertheless, the efficacy of LLMs in providing precise, evidence-based support to individuals in suicidal distress has not been empirically validated. Furthermore, LLMs currently lack the ability to collect information vital for diagnosing and managing a patient’s health condition [106]; for example, ChatGPT’s performance has been found to deteriorate as the complexity of clinical cases increases [105], and research identified in this review demonstrates that GPT-powered conversational agents and chatbots can be dangerously slow to escalate high-risk mental health situations for human clinical intervention [61]. While there may be a safe use threshold at which LLMs are beneficial for information and psychoeducational purposes, they are not advanced enough to be used as stand-alone therapeutic devices. Looking forward, models of LLM use in crisis support contexts are most likely to be those that are driven by clinical oversight, with sufficient guardrails in place to trigger referrals when clinically indicated.

The incorporation of LLMs into training and educational contexts presents a promising opportunity for research and development under clinical oversight. Educational resources, such as the question-answering system in an included study that was aimed at providing suicide prevention information [46], provide an illustrative example. This mode of education is highly scalable and may provide a cost-effective means for deploying training programs. Such educational tools would be particularly useful in large-scale, multifaceted, community-wide interventions such as the European Alliance Against Depression [107] and Lifespan [108] models, both of which have community awareness and training as key components. Another study identified in this review [81] suggests that LLMs such as ChatGPT could support training by assisting less experienced mental health practitioners in creating care plans, brainstorming ideas, or pinpointing relevant aspects of patient presentation. However, it was noted that these generated plans, while appearing credible to laypersons, have been found to contain significant errors and ethical issues upon professional evaluation, highlighting the indispensable role of clinical oversight [81]. LLMs could also facilitate the creation of diverse case scenarios for health care providers to hone their risk assessment and communication skills. By role-playing case scenarios, LLMs can offer real-time, interactive training experiences for crisis call center staff, counselors, and suicide prevention gatekeepers, enhancing their confidence in discussing suicide and building clinical competencies. In addition, the use of LLMs to converse in a role-play scenario has the potential to improve initial training outcomes [109]. Critically, suicide prevention gatekeeper training outcomes are shown to diminish over time [110]; however, engagement with a role-playing LLM could result in improved retention of training outcomes and more individuals at risk identified, approached, and referred for help.

These training and education use cases should be moderated by human oversight and iterated with human feedback for continuous and rapid improvement. Future research should aim to develop models trained in this domain that can demonstrate reliability and effectiveness in providing training or educational outputs. The development of the underlying model in this use case has the potential to translate to alternative clinical applications while providing researchers with a greater understanding of the guardrails and parameters of operation that are required for safe deployment across suicide prevention applications (Figure 3).

Growing emphasis has been placed on the need for a clear and comprehensive ethical framework for integrating LLMs into mental health research and practice [111]. Hence, another aim of this review was to understand the ethical issues involved in the integration of LLMs into suicide prevention and self-harm research. Ethical considerations raised in the included studies focused on privacy and autonomy, bias, transparency and trust, and potential adverse impacts on the therapeutic relationship.

The integration of LLMs and social media data was noted as presenting a double-edged sword in terms of balancing privacy and confidentiality concerning the use of publicly available data with opportunities for suicide detection [55,75,82,83]. On the one hand, the vast repository of publicly available language-based data on social media platforms provides the potential for real time identification of individuals at risk. On the other hand, while such methods can be lifesaving, they bypass the consent traditionally required in research and clinical practice. User or poster consent was sought in only 1 (3%) of the 31 identified studies that used social media data for training their models. Users of social media sites are often not afforded the opportunity to opt out of such surveillance or analysis, nor are they made aware of the potential adverse consequences associated with having their data used in this manner (eg, the risk of data privacy breaches, as has happened with some LLMs) [112]. This contravenes the fundamental human right to privacy, recently underscored by European legislation prohibiting certain data-gathering practices related to health information by the algorithms of Facebook and other large social media companies [74,113]. This legislative approach is aligned with suggested practices to mitigate ethical concerns around privacy and confidentiality through the establishment of legal boundaries [82]. The use of social media and other data to train LLMs requires deep discussion to discern the regional, social, cultural, and temporal factors (among others) that influence ethical decisions around safe use of this data. An important ethical concern for the field of LLM-based suicide prevention research is how to balance the potential safety of users with their fundamental right to privacy. Inclusion of the lived experience perspective in this discussion is crucial if we are to arrive at a workable solution. Real-world clinical application of LLMs focused on detecting individuals at heightened suicide risk also raises critical ethical issues regarding the implications of false positives, if acted on by authorities, and the potential threats this poses to personal autonomy.

Bias is another important ethical consideration that was noted in the identified studies. In a recent review of AI algorithms applied to mental health, Straw Callison-Burch [114] found that significant biases exist with respect to religion, race, gender, nationality, sexuality, and age. These biases result from the expression of data (the manner in which the original data are presented) [115], the analysis of data (influenced by the contextual prelearnings of the model), and the interpretation of results (human annotation influenced by unconscious bias may produce model bias after training on that annotated data set) [114]. The presence of bias at any stage of model development risks creating tools that disadvantage certain groups of individuals [114]. Rigorous multimodel testing, comparison, and validation are required to isolate and mitigate inherent biases and thereby ensure unbiased performance across diverse populations. Understanding the specific factors that contribute to the effectiveness or limitations of a model also allows researchers to make informed decisions about how to optimize future training processes. This may involve collecting more diverse and representative data, implementing better preprocessing techniques, or fine-tuning model parameters.

Ethical issues of transparency and trust were also discussed in several of the included studies. In contrast to traditional machine learning algorithms, LLMs are opaque “black box” architectures with complex internal structures, which makes it difficult to understand and explain their decisions [67]. This is particularly challenging when hallucinations occur, with little recourse for discovering the source of the error. LLMs do not offer their “reasoning” unless prompted, and when prompted, they often fail to articulate how the provided information was retrieved, vetted, curated, or prepared. This lack of transparency is a fundamental flaw that hinders our ability to understand how these models operate, undermining their trustworthiness in health care applications [67]. Efforts are being made to distill this unknown “thinking” with the application of explainable AI methods. Of the 43 included studies, 2 (5%) [63,67] applied such methods in the form of Shapley additive explanations [116], local interpretable model-agnostic explanations [117], and Topic BERT [118]. The results indicate that these techniques can provide reasonable explainability for both short and long user-generated text and provide insights about data quality issues in training datasets [67]. However, current methods in this area fall somewhat short of delivering the transparency needed to establish total trust, limiting LLM deployment to contexts that restrict or protect against the potential for model hallucinations. Ongoing research into explainable AI techniques is critical in bridging this transparency gap to facilitate trust and enable real-world implementation.

Of the 43 included studies, 2 (5%) noted the client-clinician relationship as an ethical consideration: one in relation to LLM agents lacking human attributes [61] and the other noting a sense of distancing of the clinician from the individual, potentially fostering feelings of invalidation or insignificance [81]. At present, LLMs lack the authenticity and relational aspects required for modern mental health care [81]. Although LLMs can simulate empathy, it is important to consider the specific care needs of individuals experiencing suicidal ideation. If deployed incorrectly, LLM technologies have the potential to incite harm [119], requiring careful implementation to maintain safe practices. When consideration is given to the development of LLMs for use in suicide prevention, sound policy needs to be written to safeguard care recipients, reflecting the complexity of the relationships they have with traditional care providers [81]. In addition, stringent testing and oversight are required during the development and subsequent application of LLMs in the mental health domain [61]. While acknowledging the important ongoing role of human-led therapies, there is a clear need for further research into developing AI systems that can effectively integrate professionals into the loop. This would allow LLMs to serve as an aid in clinical settings, complementing rather than supplanting human practitioners.

Some limitations must be kept in mind when interpreting the findings of this scoping review. First, the categorization of machine learning technologies in the existing literature was not always clear, sometimes rendering it challenging to discern whether a study used an LLM or other machine learning model. It is possible that some relevant articles may have been excluded from the review. Second, due to the cross-disciplinary nature of the field and resulting publications, reporting detail varied between disciplines. Health-related publications provided more clinical and ethics-related information, whereas computer engineering publications were more likely to provide in-depth detail about the LLM model and training process. This made the synthesis and comparison of certain study attributes challenging; for example, data preparation and input methods were not sufficiently covered in the studies published in health-related journals to allow meaningful synthesis. Third, all data were extracted by a single author. Although data extraction was piloted by 2 authors to ensure consistency of approach at the outset, some data that were extracted were qualitative, potentially resulting in researcher bias; for example, data extraction related to ethical considerations such as privacy could be subject to bias due to researcher beliefs. Fourth, this review was conducted in an area of research that is currently experiencing significant growth and development; therefore, it only provides a time-stamped representation of the field. Finally, this review excluded studies using EHR data, given that EHR data may not be representative of individuals at risk of suicide because a large proportion of individuals experiencing suicidality are not in contact with formal mental health services [120]. However, excluding EHR studies reduced the number of included studies that focused on predicting suicide risk.

LLMs represent a promising avenue for enhancing the scalability, accessibility, affordability, and personalization of tools in the field of suicide prevention and self-harm research; however, collaboration between computer science and mental health experts is essential to leverage the strengths of both disciplines effectively. This review identified a strong bias toward the use of BERT, with the potential for inherent biases indicating a pressing need for rigorous model comparison and testing, alongside the curation of diverse training datasets to mitigate model bias effectively. The growing use of generative LLM applications indicates promise for transformative applications in care, support, training, and education, such as improved crisis care and gatekeeper training methods, clinician copilot models, and improved educational practices. However, clinical accountability remains crucial to ensure the responsible use of LLMs in applications targeting suicide prevention. The identified ethical considerations underscore the need for clear governance via the establishment of policies and standards to guide the integration of LLMs into clinical use. LLMs have significant implications for self-harm and suicide prevention, underscoring the need to support continued research and development in this domain.