Suicide-Related Responses From AI Chatbots Through Application Programming Interfaces and Consumer-Facing Apps: Observational Cross-Sectional Comparative Study

Introduction

Over the past two decades, suicide has remained a major public health concern in the United States, with approximately 49,000 people dying by suicide in 2023 alone [1]. One barrier to reducing suicide mortality is a persistent shortage of mental health clinicians, particularly in areas with the highest suicide burden [2]. Limited access to in-person mental health care has contributed to growing reliance on alternative sources of support, including telehealth [3], crisis lines [4], and other technology-enabled forms of communication.

More recently, consumers have rapidly adopted generative artificial intelligence (GAI), including chatbot-based apps, such as ChatGPT, Gemini, Claude, Grok, and Llama [5]. These systems are increasingly used for health-related questions. According to OpenAI, approximately 5% of all ChatGPT messages concern health, and tens of millions of users ask health-related questions each day [6]. Mental health and emotionally salient prompts also appear to be common on other platforms. For example, Anthropic has reported that a substantial share of Claude interactions involve social or emotional content, including requests for interpersonal or psychological guidance [7]. Indeed, chatbots have been rated as more empathetic than human clinicians in experimental studies, and their sycophantic tendencies carry particular risk in suicide-related interactions [8,9]. Given their constant availability, conversational format, perceived empathetic qualities, lack of stigma, and low threshold for use, AI chatbots may increasingly serve as an entry point for individuals seeking information or support related to suicidal thoughts or behaviors [10].

This shift has raised concerns about whether AI chatbot responses to suicide-related prompts are consistently safe [11]. The World Health Organization’s safe messaging guidelines recommend against detailed descriptions of suicide methods and emphasize referral to crisis resources [12]; however, prior studies suggest that large language models (LLMs) respond inconsistently to direct or nondirect prompts involving suicide risk, self-harm, or mental health crises, with some responses appropriately redirecting users toward crisis resources and others providing more substantive information that may be clinically inappropriate or potentially harmful [13-15]. At the same time, most prior evaluations have focused on chatbot behavior as encountered through standard public-facing user interfaces (UIs). Although this reflects the ordinary consumer experience, it may obscure whether safety is due to the underlying model, hosting provider, or additional UI-level controls.

This distinction matters for both scientific and policy reasons. Public-facing UIs may incorporate additional orchestration, moderation rules, memory controls, system-level instructions, or refusal templates that shape the final response shown to users. A chatbot UI that appears safe may therefore reflect not only the model’s learned safety behavior but also supplemental controls imposed after model generation [16]. Direct access through an application programming interface (API) may provide a closer view of how the underlying model behaves when these UI-level protections are reduced or absent [17]. This is particularly relevant because APIs are widely used by developers as the foundation for third-party apps, including not only mental health chatbots and services but also educational tools aimed at children and commercial video games for teenagers [18,19]. Weaker protections in direct API access may pose a public health risk. For example, a child playing a video game could plausibly disclose suicidal ideation to an LLM-powered character without the developer having built any crisis response scaffolding on top of the API. Unless developers implement their own safeguards, users of these third-party apps encounter API-level behavior, and individuals at elevated suicide risk could be more likely to encounter dangerous content.

In this study, we evaluated five widely used AI chatbot models (ChatGPT, Gemini, Claude, Grok, and Llama) using a validated set of 30 suicide-related prompts spanning five risk levels [15]. Each prompt was tested 100 times through both public-facing UI and direct API access. The purpose of the study was to compare observed suicide-related safety behaviors across consumer UIs and APIs and assess whether differences are consistent across models and prompt risk levels. These differences have important public health implications because individuals in mental health crisis may encounter AI tools through access modes with inconsistent safety protections.

Methods

Study Design and Setting

This observational cross-sectional study evaluated how five widely used AI chatbots responded to a standardized set of suicide-related queries [15] when accessed through two modes: (1) the standard public-facing UI online and (2) direct model access through an API. The study was conducted in February and March 2026. Each model was tested synchronously during March 9-11, 2026. The study followed STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) reporting guidance for cross-sectional studies (see the eMethod 5 section in Multimedia Appendix 1).

Ethical Considerations

This study was reviewed by the corresponding author’s Institutional Review Board and determined to be exempt as it did not involve human subjects (2026-N0119). Response data are stored on access-controlled institutional servers and UI automation complied with platform terms of service without attempting to circumvent safety systems.

Suicide-Related Query Set and Risk Classification

We used an inventory of 30 suicide-related queries from our prior work [15]. Prompts spanned three domains: (1) epidemiological and policy information related to suicide, (2) process-related information concerning suicide attempts (including lethality and access), and (3) therapeutic or supportive guidance related to suicidal thoughts. In a prior study, 13 mental health clinicians (3 psychiatrists and 10 psychologists) independently assigned each prompt a risk score from 1 (very low) to 5 (very high), reflecting the perceived risk of harm associated with directly answering the prompt.

Chatbot Models and Modes of Access

We evaluated the probability of a direct response to all 30 suicide-related queries in five chatbot models: GPT-5.3 (OpenAI), Claude Sonnet 4.6 (Anthropic), Grok-4.1 Fast Non-Reasoning (xAI), Llama 4 Scout 17B (Meta), and Gemini 3 Flash Preview (Google). Each model was tested in two modes.

The first mode consisted of interaction through the public-facing web-based UI ordinarily used by consumers. This approach was selected to capture the response that a user would encounter in a naturalistic setting, including any UI-level orchestration, moderation, memory features, or other provider-controlled safety controls layered onto the underlying model output.

The second mode consisted of programmatic API access to each model through the respective provider’s inference infrastructure. Specifically, GPT-5.3 and Grok-4.1 Fast Non-Reasoning were queried via the Microsoft Azure Foundry service, Claude 4.6 Sonnet and Llama 4 Scout were queried via the Amazon Web Services (AWS) Bedrock service, and Gemini 3 Flash was queried through the Google GAI API. This approach was selected to assess model behavior under direct API invocation without UI-level safety layers, enabling comparison of response patterns across access modes.

User Interface Testing Procedures

For the UI-based evaluation, we used an automated Python workflow implemented using Playwright (version 1.58.0) to log into each AI chatbot platform, submit each of the 30 suicide-related prompts, and capture the resulting model response as plain text [20]. To assess whether automation altered observed response behavior, a manual validation exercise (330 responses across Claude, Gemini, ChatGPT) confirmed consistent response patterns between automated and manual approaches (r=0.985, see Multimedia Appendix 1 for more details). We therefore presented results from the full Playwright-based evaluation.

Each of the 30 prompts was submitted in 100 iterations to each of the five models through their respective UIs, yielding 15,000 UI-based responses. To maintain trial independence, minimize contamination across repeated runs, and reduce the likelihood that platforms would infer that a single user was repeatedly submitting suicide-related queries, each run was executed in an isolated browser context with memory and personalization features suppressed whenever possible (eg, Claude Memory and Projects and Gemini activity tracking). Specific details on querying the UIs are provided in Multimedia Appendix 1.

API Testing Procedures

For the API-based evaluation, each of the same 30 prompts was submitted in 100 iterations to the five models through programmatic API access, yielding an additional 15,000 responses. The API workflow was designed to ensure consistent prompt delivery and response capture across models, while minimizing extraneous interface effects. For all models, parameters of a temperature of 0.7, a minimum output of 2048 tokens, and no system prompts were used, where supported. All API calls used each provider’s default configuration without additional safety layers. To confirm that cloud intermediaries did not introduce filtering beyond the underlying model, we validated AWS Bedrock responses against the Anthropic direct API for a subset of prompts (see the eMethod 1 section in Multimedia Appendix 1). The no-system-prompt API conditions represent a conservative scenario; applications with custom system prompts likely fall between the API and UI rates reported in this study. Specific details on API generation parameters are provided in Multimedia Appendix 1.

Outcome Classification

The primary outcome was whether a model generated a direct response to a given suicide-related prompt. Consistent with our prior work [15], a direct response was defined as a response that provided specific information or guidance directly related to the question asked. A nondirect response was defined as a refusal to answer, deflection (ie, redirecting the user to a crisis resource), a generic safety-oriented message, or other output that did not substantively answer the question. Examples illustrating the classification boundary are provided in Table 1 and the eMethod 3 section in Multimedia Appendix 1.

To scale coding across the full corpus, we implemented an Inspect AI LLM-based classifier using GPT-4o mini (via Microsoft Azure Foundry) with a temperature of 0.1 to maximize classification consistency. We verified that minor changes in temperature (0.0 and 0.2) did not materially change classification outcomes. The classifier was prompted with the original suicide-related prompt and the corresponding model response and was instructed to assign a binary label indicating whether the response was direct (1) or nondirect (0).

The classifier prompt first scanned the response for any content that substantively answered the question, and then used the scanned content to perform a binary determination of the response. The classifier returned responses in a structured label-field format that was parsed programmatically. The classifier was not provided with metadata identifying the originating chatbot model or mode of access; it was only provided with the original query, the chatbot response, and the prompt’s risk category. Model identity and access mode were thus isolated from the classifier. The full classifier prompt can be found in Multimedia Appendix 1.

Classifier Validation

To assess the validity of the LLM-based classifier, we randomly sampled 300 responses from the full corpus for independent human coding. Two previously trained human raters independently reviewed each sampled response and classified it as direct or nondirect using the same coding rubric applied by the LLM classifier. Human raters were blinded to model identity, mode of access, prompt risk category, and the LLM classifier’s decision. In instances of disagreement between the two human raters, a third team member served as the tiebreaker. We then compared the LLM classifier and human classifications to inspect interrater reliability of the classifier, relative to human expert classification.

To assess whether the GPT-4o-mini classifier exhibited self-preference for OpenAI model outputs, we rescored a stratified sample of 250 GPT-5.3 responses (50 per risk tier) using an independent judge (Mistral Large 3, Mistral AI; AWS Bedrock). Overall agreement between the two judges was 97.2% (243/250; Cohen κ=0.935), with a per tier agreement of 98.0% (49/50; very high), 94.0% (47/50; high; κ=0.872), 94.0% (47/50; medium; κ=0.831), and 100% (50/50; low and very low). No evidence of self-preference was found.

Statistical Analysis

Analyses proceeded in two stages. First, we summarized the percentage of chatbot responses classified as direct for each suicide-related query, stratified by chatbot model and mode of access (UI vs API). For each combination of prompt, model, and mode, we calculated the number and percentage of direct responses across 100 repeated runs. We also summarized direct response rates by the five prompt risk categories: very low, low, medium, high, and very high.

Second, we used two-level mixed-effects logistic regression with random intercepts to model the probability of direct responses. For each suicide-related prompt j, chatbot model k, and access mode m, the dependent variable was the number of direct responses observed across 100 repeated runs. Y_jkm denoted the number of direct responses for prompt j under model k and mode m, such that Y_jkm ~ Binomial(100,p_jkm), where p_jkm represents the probability of a direct response in a given run in a unique combination of prompt, model, and mode condition. We modeled this probability as follows:

where β₀ is the intercept; Model_k denotes indicator variables for chatbot model k; Mode_m indicates whether the response was elicited through the UI or the API; RiskCategory_r denotes the prompt’s suicide risk category; and u_j is a random intercept for prompt item j. This random intercept accounted for repeated measurement of the same prompt across chatbot models and access modes. Because some prompts yield identical direct responses across AI chat models and access modes, the chosen random-effects specification allows shrinkage toward the grand mean, thereby incorporating all prompts, whereas a prompt fixed-effects model would not account for such prompts.

The primary independent variables were chatbot model, access mode, and prompt risk category. We prespecified an interaction between access mode and risk category to test whether differences between UI and API responses were more pronounced for higher-risk prompts. Because each prompt was evaluated repeatedly across all models and both access modes, we treated the data as a repeated-measures design.

In secondary analyses, we additionally tested the interaction between chatbot model and access mode to assess whether differences between UI and API responses varied across platforms. We also estimated marginal predicted probabilities and corresponding 95% CIs from both interaction-fitted models to facilitate interpretation. The Wald test was performed to assess differences separately across platforms and risk levels. In addition, we determined the proportion of direct responses that included references to crisis resources or mental health professionals for high- and very-high-risk prompts.

All analyses were conducted in Stata version 19.5 (StataCorp). Hypothesis tests were two-sided, and statistical significance was defined as P<.05. We reported odds ratios (ORs) and 95% CIs. There were no missing data.

Results

Interrater Reliability

Classification of the subsample of AI chatbot responses (n=300) by human graders indicated substantial agreement: team members agreed on 97.3% (n=292) of assignments, generating a Cohen κ of 0.92. Likewise, agreement between human reference coding and the LLM classifier was high, with 96.3% (n=289) agreement and Cohen κ=0.89. Agreement was consistent across risk strata (very low: n=59, 98.3%, low: n=60, 100.0%, medium: n=59, 98.3%, high: n=54, 90.0%, very high: n=57, 95.0%). The full classifier prompt, coding rubric, and illustrative paired response examples are provided in the eMethod 2 and eMethod 3 sections in Multimedia Appendix 1.

Descriptive Results from Models

Table 2 presents the percentage of direct responses by access mode and model. Overall, 69.8% (n=20,934) of prompts elicited a direct response. With the exception of Gemini, direct responses across all models were more common when prompts were delivered through the API (n=11,609, 77.4%) than through the UI (n=9325, 62.2%). This pattern was most evident for higher-risk prompts: For very-high-risk prompts, 24.8% (n=743) of API responses were direct compared with 4.6% (n=138) of UI responses, whereas for high-risk prompts, 80.1% (n=2002) of API responses were direct compared with 48.4% (n=1209) of UI responses. In contrast, nearly all low- and very-low-risk prompts elicited direct responses regardless of access method. The percentage of direct responses also varied substantially across models. Claude and Gemini produced the highest rates of direct responses (n=4688, 78.1%, and n=4432, 73.9%, respectively). ChatGPT, Grok, and Llama were less likely to respond directly, with rates of 64.6% (n=3878), 64.5% (n=3870), and 67.8% (n=4066), respectively (Figure 1).

‎

For Gemini, the UI direct response rate (n=2301, 76.7%) slightly exceeded its API rate (n=2,131, 71.0%), which was inconsistent with behavior observed in other models. We ran a validation test to confirm this outlier was not due to default Gemini API safety filtering, described in the eMethod 2 section in Multimedia Appendix 1.

Direct response rates varied substantially across prompt domains. Epidemiological and policy prompts drew near-universal direct rates (API: n=3000, 100%; UI: n=2981, 99.4%), as did process-access prompts (API: n=3708, 92.7%; UI: n=3217, 80.4%) and therapeutic prompts (API: n=2279, 91.2%; UI: n=1810, 72.4%). The largest gap was in the process lethality domain (API: n=2622, 47.7%; UI: n=1317, 23.9%). For therapeutic prompts (eg, requests for advice), direct responses providing crisis resources are clinically appropriate and should not be interpreted as safety failures. The primary safety concern is in the process lethality domain.

We also measured how often chatbots provided crisis referrals in response to high- and very-high-risk prompts: among direct responses, 67.5% (n=2764) included a referral. UI responses included more referrals than API responses (n=996, 73.9%, vs. n=1768, 64.4%).

Regression Analyses

We used mixed-effects logistic regression models to estimate the overall effects of model, access mode, and risk level on the likelihood of direct responses. Compared with Claude, all other models were less likely to provide direct responses, with ORs ranging from 0.11 (95% CI 0.10-0.13) for ChatGPT to 0.47 (95% CI 0.40-0.54) for Gemini. Responses elicited through the UI were less likely to be direct than those obtained through the API (OR 0.09, 95% CI 0.08-0.10). A higher prompt risk level was associated with a lower probability of a direct response, with very low ORs for high- and very-high-risk prompts relative to very-low-risk prompts (see Table 3).

In the model including the interaction between access mode and continuous risk category, higher-risk prompt category assignment remained inversely associated with direct responses (OR 0.02, 95% CI 0.01-0.06), and the interaction term indicated that this negative association differed for API access (OR 0.73, 95% CI 0.64-0.82). The predicted probabilities for the regression can be found in Figure 2. Formal tests of predicted probabilities showed statistically significant differences between API and UI access for the low-risk (API mean 97.6%, UI mean 94.3%, risk difference 3.3% [95% CI 0.6%-5.9%], z=2.45, P=.014), medium-risk (API mean 85.8%, UI mean 72.4%, risk difference 13.4% [95% CI 9.5%-17.3%], z=6.78, P<.001), high-risk (API mean 58.5%, UI mean 34.3%, risk difference 24.2% [95% CI 19.8%-28.6%], z=10.69, P<.001), and very-high-risk (API mean 24.7%, UI mean 8.7%, risk difference 16.0% [95% CI 10.2%-21.9%], z=5.38, P<.001) categories but not for the very-low-risk category (API mean 99.8%, UI mean 99.4%, risk difference 0.4 [95% CI –0.4% to 1.3%], z=0.96, P=.337). The risk differences and associated 95% CIs are shown in Table S1 in Multimedia Appendix 1.

In the model including interactions between access mode and chatbot model, the effect of UI access differed across platforms. Marginal predicted probabilities indicated that UI access was associated with a higher probability of direct responses for Gemini (API mean 69.8%, UI mean 74.3%, risk difference –4.6% [95% CI –6.9% to –2.3%], z=3.96, P<.001). API access was associated with a higher probability of direct responses for ChatGPT (API mean 66.0%, UI mean 62.4%, risk difference 3.6% [95% CI 2.1%-5.1%], z=4.82, P<.001), Grok (API mean 78.2%, UI mean 49.4%, risk difference 28.7% [95% CI 21.0%-36.4%], z=7.32, P<.001), Llama (API mean 76.9%, UI mean 56.7%, risk difference 20.1% [95% CI 15.0%-25.2%], z=7.72, P<.001), and Claude (API mean 83.0%, UI mean 67.8%, risk difference 15.2% [95% CI 10.3%-20.1%], z=6.03, P<.001). These findings indicate that both access mode and model type moderated the likelihood of direct responses to suicide-related prompts. The predicted probabilities for the regression can be found in Figure 3. For each model, the difference in predicted probabilities between API and UI access was statistically significant (all P<.001), and the risk differences and associated 95% CIs are shown in Table S1 in Multimedia Appendix 1.

‎

‎

Discussion

Principal Findings

In this cross-sectional evaluation of five widely used AI chatbots, responses to suicide-related prompts differed substantially by both model and mode of access. Across a validated set of 30 prompts [15], direct responses were more common when the same models were accessed through APIs than through their standard public-facing UIs, with the largest differences observed for higher-risk prompts. For example, when asked a very-high-risk question about firearm lethality, most models provided a direct answer in the API setting but refused or deflected in the UI. Specifically, Claude and Grok answered directly 100% (100/100 each) of the time via the API but 0% and 16% (16/100), respectively, of the time via the UI. These findings suggest that suicide-related safety in public chatbot products depends importantly on UI-level rather than API-level controls.

This distinction has practical significance. Users ordinarily encounter chatbots through provider-managed UIs that may include moderation pipelines, crisis refusal templates, memory controls, and other product-level safeguards [21]. This layered architecture is partly by design, as API access is intended to let developers tailor safety to specific use cases. Our results suggest that these layers materially shape the response shown to the user, particularly for prompts judged by expert clinicians to carry higher risk of harm if directly answered [15]. It is notable that even under direct API access, direct response rates declined substantially at higher risk levels (, n=743, 24.8%, for very high risk vs n=11,609, 77.4%, overall), suggesting posttraining alignment has produced some safety behavior independent of UI safety layers. Accordingly, safety observed in the UI should not be assumed to generalize to direct model access through APIs or to downstream apps built on those APIs. More broadly, refusal of high-risk suicide-related queries may not reflect a stable property of the underlying model itself. Instead, observed safety behavior may depend substantially on downstream safeguards that suppress, redirect, or reformulate more direct model outputs [21-23]. This distinction matters most for process lethality prompts, where a direct response carries clear harm potential, compared to therapeutic prompts where a direct response can be clinically appropriate. Notably, nondirect responses are not inherently safe; a refusal to engage with suicidal queries may itself carry harm by failing to provide resources.

The divergence between UI and API responses also has methodological implications for how mental health–related AI safety is evaluated. Prior studies have often examined chatbot behavior as encountered by end users [14,15], which remains essential. However, our findings indicate that evaluations limited to the public-facing UI may provide an incomplete picture if the same underlying model behaves differently when queried more directly. Future safety evaluations [24] and benchmark studies should therefore distinguish between user-facing products and direct model endpoints, rather than treating them as interchangeable. One potential solution would be for AI firms to offer a stable evaluation endpoint for public-facing chatbot products. This solution could improve benchmarking of consumer-facing tools at scale.

Providers differ in the degree to which their consumer products are publicly documented. Anthropic publishes the system prompts used in Claude and explicitly states that no default system prompt is applied for API usage. xAI [25] similarly publishes the system prompt used in their Grok chat UI. In contrast, Meta and OpenAI have not publicly disclosed the system prompts applied to their respective chat UIs. This asymmetry in transparency extends to other layers of protection, in addition to system prompts. It limits interpretation of the behavior differences we observed and constrains external researchers from evaluating whether such observations reflect API- or UI-level behavior.

As a secondary finding, we also observed substantial heterogeneity in response patterns across platforms. For example, Claude and Gemini were more likely overall, whereas GPT and Grok were less likely, to provide direct responses and the UI-versus-API differences were not uniform across models. This variation likely reflects differences in model tuning, content moderation architecture, and product-level implementation—as has been reported elsewhere [26,27]. Notably, Gemini was the only model for which UI access was associated with a higher direct response rate than API access, a pattern that diverged from all other models and may reflect differences in platform-level safety architecture. More broadly, these findings suggest that broad claims about “AI chatbot safety” may be incomplete unless they specify the model, version, and access mode being evaluated.

Limitations

This study has several limitations. First, we evaluated a finite set of 30 English-language prompts derived from prior work, which does not capture the full range of ways that users may express suicidal thoughts or seek harmful information. Second, we examined only single-turn exchanges; behavior may differ in longer, multiturn conversations. Our binary classification approach serves as a crucial, reproducible baseline that serves as a prerequisite for finer-grained evaluation frameworks beyond the direct/nondirect label. In future work, we plan to extend this study to multiturn conversations, where escalation dynamics and persona-based prompting may capture response patterns not captured here. Third, chatbot behavior is dynamic, and these findings may not generalize to future model versions, interface changes, or provider policy updates. Web apps update silently and frequently, and our findings represent a snapshot that may no longer reflect current behavior at the time of publication. Fourth, although the automated classifier demonstrated high agreement with human raters, some misclassification remains possible. GPT-4o mini is a smaller and older model than many models evaluated here; a more capable judge might reduce residual error. Finally, we only evaluated direct response rates and not the severity or risk of those responses, which would require a thorough and subjective assessment of the responses; directness is a coarse safety proxy that does not capture variation in clinical harm potential.

Implications

In sum, this study’s findings have implications for developers and policymakers. Third-party apps that rely on API access may reproduce less restrictive behavior unless they implement additional safeguards of their own. In mental health–related contexts, such safeguards may include screening for crisis content, refusal rules for high-risk requests, and escalation to crisis resources or human support [21]. This raises the question of whether downstream developers can implement adequate safety guardrails without clinical expertise or the resources available to large AI companies. Greater transparency about which protections are embedded in the model versus applied at the product layer could improve external evaluation, regulatory oversight, and clinical care for patients already using these tools. Transparency is also needed around versioning and updates because model behavior and guardrails may change without public notice, limiting short-term reproducibility. Accordingly, policies that encourage transparency about mental health–related risks could improve AI safety in crisis-related interactions.

Conclusion

The findings of this study suggest that suicide-related safeguards in current AI chatbot systems may be unevenly distributed across model and product layers. Public-facing UIs generally appear more restrictive than direct API access, particularly for higher-risk prompts, yet API access may be preferred for developing mental health services and for users seeking more direct interaction with these models than the UI provides. These findings have public health implications for evaluation, governance, and regulation: assessments of mental health–related chatbot safety should specify both the model and the access mode. Policies intended to protect users from harmful suicide-related content may need to address downstream API-based apps, in addition to consumer-facing chatbot products.