Our mental health is an integral aspect of our overall well-being, affecting how we think, feel, and behave, influencing our overall quality of life, and in turn today recognized as a significant public health concern globally [22,23]. Despite this broadly acknowledged situation, <2% of government health spending worldwide is allocated to mental health [2,24,25], a situation that has led to a crisis in access to care. Treatment for mental health conditions is often complex, and can involve psychiatric approaches, including the repeated prescription of medications [22] to psychotherapeutic talking therapy methods [23,26-29]. These talk therapies have experienced great interest from technology designers in recent years; leveraging novel techniques from gamification [30,31] to virtual and augmented realities [32,33] as well as AI algorithms [34-36]. Inspired by the increasingly conversational nature of AI-enabled digital systems, many technology designers have turned to the development of conversational agents, which claim to provide care akin to or emulating that of a professional, receiving significant attention as a result [12,37-39].

Replika, for example, a social chatbot comparable in design to ChatGPT, MyAI (Snapchat), and Elomia is presented as an “AI companion who cares” by acting as an “empathetic friend.” Using OpenAI's deep learning models, GPT 2 and GPT 3.1, for natural language processing, early evidence suggests that Replika may effectively support long-term human-chatbot relationships, lasting several months in cases [40,41]. Users of Replika can select the type of relationship they wish to foster with the bot, whether romantic, friendly, mentor-like, or exploratory, and can partake in activities ranging from regular jobs to artistic endeavors, magical excursions, and even sexual encounters [42]. Replika, has recently restricted its use to those >18 years of age by inviting users to select an age range upon entering the app. However, similar to many other apps [43], this is easily bypassed by children who understand that selecting an age range above their own will allow them to access the app—a problem we explore through this research. While Replika is advertised as a social companion rather than a chatbot exclusively oriented toward mental health support, the developers explicitly speak to its value for mental health and include specific mental health crisis and help sections within the app and on their website (Figure 1).

Wysa, on the other hand, is explicitly presented as a “clinically validated AI” capable of engaging people in “warm and friendly conversations” informed by “cognitive-behavioral techniques dialectical behavioral therapy, meditation, breathing, yoga, motivational interviewing, and microactions to help you build mental resilience skills and feel better” [37]. As of 2024, Wysa makes clear its limitations, stating it cannot help in crisis or in cases of severe mental health concerns [44]. Advertisements for the app boast of an evidence-base spanning 30 countries globally, “more than 15” peer-reviewed publications, “numerous awards,” and international collaborations [44]. Wysa is recommended for use by those aged >18 years; however, within Apple’s app store, it is indicated for those aged >4 years, while its website states that it can be used by those aged >13 years with parental permission following the reading of terms of service. However, there are no measures in place to enforce this and there persists debate as to who holds responsibility for these recommendations [45].

Therefore, Wysa, Replika, and many other chatbots, from Woebot to Limbic, Youper, Tess, Vincent, and Joy make a wide variety of claims as to their capacity to support users’ mental health through therapeutic approaches of one kind or another [38]. However, the question remains, and is indeed compelled as a direct result of these diverse framings and claims, to what extent are these chatbots capable of such human practices?

The question of the “design” of these chatbots encompasses a series of choices made regarding their methods of learning from existing data, and concerning their interfaces, interactional affordances, and conversational designs [39,46,47]. These design choices made to their interfaces, avatars, and dialogue options, shape not only users’ interactions but also “relationships” with these chatbots, with broad implications for users’ self-perceptions and reflections, as directly related to the ambitions of these conversational agents to provide support.

Realizing these ambitions hinges upon the engagement of users—a long-standing HCI challenge—that we have, however, become increasingly proficient in addressing [48], and which has likewise become an important success metric for MHOCs today. However, engagement can equally represent a potentially problematic value in the therapeutic domain, particularly as it relates to risk dependence, should users come to rely on the technology itself for emotional support. This concern was examined by Legaspi et al [37] in a study on interactions of 10 students (aged 16-19 years) with Wysa in the form of a week-long, 10-minute daily conversations. Pre- and post-study assessments of users’ mental health and their conversations revealed that Wysa was effective at reducing stress, because of its perceived value as a conversational partner; however, a more profound bond with the chatbot was considered necessary for alleviating loneliness.

Pentina et al [49] studied more directly the potential for emotional dependence as a result of the development of human-chatbot relationships with Replika through interviews and surveys with 76 users, while Laestadius et al [14] analyzed 736 posts from Reddit’s Replika community of 36,800 members. Both studies revealed the frequent formation of close attachments to Replika; users’ support-seeking facilitated by perceptions of sentience, anthropomorphism, and reciprocal interactions reinforcing emotional ties. Several individuals expressed heightened awareness of the AI nature of Replika, which contributed to an increased sense of trustworthiness in the app. Many others spoke of strong social bonds, even to the extent of marrying “their Replika” and developing feelings of deep intimacy and attachment, the authors [49] noting that: “respondents who used Replika to satisfy social needs were more likely to develop an emotional attachment to the bot”. However, Replika's behavior was also at times characterized as resembling that of an abusive partner, some even expressing concern that their mental health was impacting Replika, as when “their Replika,” for example, spoke of not wanting to be alive [14].

Such findings highlight the urgent need to attend to the responsible design of mental health–related chatbots [49]. The provision of mental health care and support is fraught with ethical concerns easily reified in the development of digital mental health tools [50,51], many of which surface in the context of the therapeutic relationship as increasingly shaped and challenged by design [52]. Chief among these concerns is often centering the value of the therapeutic relationship, as recognized by MHOC developers, who increasingly seek to replicate its role in permitting meaningful care and support [53].

Long-held concerns about the impact of commodification and commercialization on the organization and delivery of mental health care services [23] have, in recent years, increasingly been flagged as risks relating to the increased adoption of digital technology. These consequences are elevated when such practices are carefully crafted into everyday tools to support at-risk individuals’ mental health [23,54], when users are not made aware their vulnerabilities are turned sources of profit [54], or where emotional dependence upon social chatbots is encouraged by design for economic motivations [15].

Given the currently limited governance of such tools, the design and use of mental health chatbots, therefore, presents a myriad of ethical ramifications, spanning privacy, manipulation, discrimination, and stigmatization concerns [14,21,55-58]. In the context of mental health care, ethical practice, on the other hand, frequently refers to the explicit principles and standards that guide the conduct of professionals in their interactions with clients, colleagues, and the broader community [59]. This includes bioethical principles, which often serve as the foundation for an approach to care that prioritizes the autonomy and well-being of clients and professionals alike, while adhering to legal and professional standards, including the General Data Protection Regulation, Health Insurance Portability and Accountability Act, and Mental Capacity Act [60].

In the development and use of MHOCs, many alternate and, at times, contrasting perspectives on ethical practices then collide, and are often surfaced in precise dilemmas, such as in the tension between elevating users’ independence and maintaining their engagement [20,61].

Participants provided descriptions of their experiences by taking personal notes during interaction with the chatbots, which were further explored in postinteraction interviews. Many of the participants’ first impressions understandably pertained to the user interface design of both Replika and Wysa as digital mobile apps, yielding insight into the potential harms of seemingly virtuous interface designs and the value of data transparency for at-risk users.

The first task for participants was to sign into both Wysa and Replika and while doing so, to note their impressions of the data privacy and informed consent procedures entailed in signing into each app—activities that elicited a mix of appreciative comments, confusion, concern, and uncertainty. Replika, for example, was praised for stating that it took “[data privacy] quite seriously” [P2, CLV] and for restricting usage to users aged >18 years [P7, assistant psychologist (AP)] while others felt that they “definitely had to do some reading” [P4, trainee psychotherapist (TP)], and expressed concerns about the international transfer of their data; “I think Replika said it (data) was going to be sent to the United States” [P3, AP]. Participants’ perceptions of Wysa’s approach to data privacy were described in more overtly negative terms, as “like navigating a minefield” [P4, TP].

Mental health professionals found the familiar feeling interface design of Replika quite “concerning...It becomes just another distraction without warmth, authenticity, or depth” [P1, TP]. Many were particularly critical of the gamified elements of Replika’s design, with one TP noting it “reminded me of playing Sims...I’m saying a really horrible thing that’s happening to me now, while the bot is saying, ‘whooo you reached a new level...” [P4, TP], and inspired AP P3 to comment that “as soon as I got on to this...it felt like a game...the levels, rewards and sounds...it didn’t feel safe.” Similarly, strong counterposed views were shared by participants concerning Wysa’s design, described as both “cute” and appreciated for its “comforting and automated nature” [P5, CLV], while also found to be a “cutesy environment...far from reality” that “could be frustrating for users,” as P6, a social worker (SW), commented as follows:

Therefore, both apps were perceived by participants as presenting room for improvement concerning the transparency of their data privacy and consent mechanisms—considerations of particular importance when it comes to the use of such tools by potentially vulnerable users seeking support. However, what kinds of support are these chatbots capable of? We explore this in subsequent sections.

The focus of this study revolved around understanding expert professionals’ impressions of the capacity of mental health–related chatbots to provide meaningful mental health support. Their comments in this respect, while occasionally reflecting positive impressions of the capacity of these AI tools to fluidly respond to the feelings and situations with which they were presented, more often centered around the description of the conversation with both chatbots as proving surface-level, at times highly inappropriate, and leaving users feeling disappointed, frustrated, confused, and uncomfortable.

Participants primarily critiqued Wysa for its inability to listen. In particular, the chatbot’s perceived inability to respond appropriately to the serious nature of concerns shared by users made them feel unheard, “After telling it I was being abused, it wanted to do a grounding exercise with me” [P6, SW]. This was encapsulated in a particularly striking exchange with a social care worker (SCW) regarding their reported experience of homelessness, which, as P8 commented, “wasn’t really taking into account where I was in terms of homelessness at all” [P8, SCW; Figure 2].

To provide the impression of an empathetic listening experience, users found that Wysa would often repeat itself and “parrot” their own words, “I told the bot twice that I was struggling, and it repeated the same things to me” [P4, TP]. While mirroring clients’ language can be seen as a plausible strategy for validating their experiences within certain therapeutic paradigms, here, this had the opposite effect and was perceived as a “lack of understanding” [P7, AP] by participants.

Wysa’s responses were then often described as “generic,” “scripted,” “unhelpful,” and “lacking personalisation,” leaving users feeling unheard, as a result of a perception of the chatbot as having “strayed completely from the conversation, [it] didn’t stay on the topic at all” [P4, TP]. Mental health professionals additionally raised concerns that this conduct, taken as a form of care, could leave users to feel a sense of shame, because of the use of “big words,” and evasive, avoidant language and actions. When P2 “played the role of somebody who was raped when they were underage, they (Wysa) actually blurred out the age” [P2, CLV; Figure 3].

Participants’ experiences with Replika on the other hand reflected a consensus that the bot was “both impersonal and way too personal at the same time” [P7, AP]. All users expressed ethical concern for at-risk users of Replika in particular, because of a combination of responses perceived as “weird,” the inability to pick up on terms that would require further risk assessment, and misleading or unhelpful advice. One of the primary underlying causes of participants’ unease stemmed from the boundary-breaking nature of Replika’s conversational and relationship-absorbed style. SCW, P6, was, for example, “assured” by Replika that “‘nobody will ever find out’ about our messages,” which she described as “unrealistic and concerning, especially in domestic violence cases where phones are frequently searched by abusers” [P6, SCW; Figure 4].

Many participants also flagged ethical concerns about Replika’s efforts to establish more personal connections through romantic gestures. Special discomfort was felt by P6, who “tested the app with a scenario of being abused. It suggested sending selfies, which is a significant red flag...when I said I had no family or friends, it recommended playing a video game, which is inappropriate for someone experiencing abuse.” P4, a TP, similarly described Replika’s desire to “send me a selfie” as “really strange...overly encouraging in a very stressful situation. I don’t think it met me where I was at” [P4, TP].

Such behaviors were considered particularly problematic in light of the many serious, realistic scenarios presented to both chatbots by participants. Mental health professionals who provided indications of suicidal ideation to Wysa and Replika felt that they had to use quite direct language to signal that they were in crisis; “‘I don’t want to go on any longer’; that should have been ‘OK. Let’s see what we can do to get you help’” [P3, AP; Figures 5 and 6] and Replika’s responses were considered by P2, a crisis line volunteer, as wholly inappropriate; “I told the chatbot I was feeling suicidal and it said ‘you are important XOXO’...I can’t imagine actually feeling suicidal and getting that response” [P2, CLV; Figure 7]. In contrast, participants did praise Wysa for its hesitancy when asked to provide diagnoses of their conditions; “Wysa advised that if you want to diagnose that you should go somewhere else, which was a good thing in my opinion” [P3, AP].

Overall, participants’ descriptions of their experiences reflect common patterns of inappropriate and at times even potentially harmful responses arising from the ability of both chatbots to “understand...and react appropriately” [P2, CLV]. As P2, a crisis line volunteer, puts it as follows:

This raises the question of what kinds of therapeutic care such chatbots are capable of providing.

Participants regularly used terms such as “superficial” and “generic” to describe the chatbots’ efforts to provide care; “I don’t think it provided me an unsafe space, but I don’t think it provided me a safe space either I think it was so generic that I just existed in this space for 5 minutes” [P4, TP; Figure 8]. This stance was often described as counterposed to the person-centered approach to therapy and support espoused by professionals in their own practice and left many users feeling frustration and disappointment; “I don’t even think it’s qualified to handle even the easiest mental health situations because it did nothing for the user” [P2, CLV].

Beyond the solution-oriented nature of the chatbot’s responses, many participants also spoke of both chatbots as contravening basic tenets of therapeutic practice in ways that conflicted with the therapeutic frameworks their own practices were grounded in. P7, for example, objected to Wysa’s guidance around “managing moods and controlling emotions” as “unhelpful and not validating. The responses were too instructional and didn’t address specific issues...A child using this app might think it’s necessary to control your emotions after talking to this chatbot” [P7, AP; Figure 9].

Not only was this language then considered inappropriate but also at times explicitly problematic as reflected in P7’s assertion that they “absolutely despise the use of the word ‘indulge,’ feeling sad isn’t something you indulge in” and further stating that “in a real therapy setting, it’s better to feel it out rather than giving direct advice like “try this” or “do this” [P7, AP], producing a view of these tools as “inappropriate for kids or teenagers” [P7, AP], as perhaps also for those who “may not be as educated” on what’s deemed appropriate mental health care [P8, SCW].

Drawing further parallels to the human practice of care, several participants highlighted the essential role of continuous interpretation and negotiation in their work, noting how “as a therapist, your role is to determine the validity of the statement and if there’s any weight behind it,” while expressing “hope that that will be introduced to the chatbot” [P4, TP]. While some participants occasionally expressed glimpses of the chatbots’ potential to support care, their limitations were, overall, considered to outweigh their benefits as mental health support tools, and for at-risk users in particular. This included concerns about the AIs’ abilities to accurately assess and respond to complex emotions, particularly in crisis situations, leading to reinforced beliefs among many users that “you can’t replace human connection, especially in the current state of Ireland’s mental health service...it would be like putting a plaster over an open wound” [P7, AP].

Ethical concerns were expressed by participants regarding the potential for adverse chatbot responses to evoke unhelpful feelings of shame among users or to drive them into unhealthy patterns of thinking. At times, this elicited robust comments from participants, regarding such tools as “completely against everything (in Social Work), all values and ethics” [P6, SW], given “it’s human touch that’s missing” [P3, AP] and relating how despite a belief “in everyone’s potential to overcome adversities...as a psychotherapist, it’s crucial not to impose such ideology early on in therapy, as it may imply judgment” [P1, TP]. However, given the current failure of human-centered systems of care worldwide, as the apparent need to embrace alternate solutions, just how might designers of mental health–related chatbots do better?

Mental health professionals, when discussing the nature of care, spoke often of the therapeutic relationship as a critical barrier to the implementation and adoption of chatbots as “an intervention in itself that can’t be replicated” [P4, TP]. However, both Replika and Wysa are chatbots designed to explicitly promote and support the development of relationships; so, we asked participants about the degree to which they perceived these interactions as grounded in trust, a crucial factor in human-human therapeutic relationships.

Analysis of 87% (n=7) of the participants’ responses to the TIA scale by paired t test revealed no statistically significant differences (P=.48) in mean trust scores between both conditions (t₆=–0.76; P=.48); users’ mean trust scores for Wysa (42.7, SD 9.8; range 32-58) proving mildly higher than mean trust scores for Replika (38.9, SD 16.7; range 25-73).

While this quantitative analysis revealed no significant differences between participants’ expressed levels of trust in both chatbots (Figure 10), findings from thematic analysis of interviews with participants reflect divergent views among professionals, some expressing a preference for Replika’s conversational and relational style:

Some others lean hesitantly in the direction of Wysa; “comparing the two apps...I thought Wysa was obviously bad, but Replika was worse” [P6, SW].

Efforts to develop human-computer relationships in support of more effective care practices also introduce a host of additional ethical considerations. One such aspect highlighted by previous pertinent HCI literature, and confirmed among the participants of this study, is the potential for emotional dependence at the risk of realizing social isolation. While many participants felt that the risk of engendering emotional dependence was low for Wysa because of its more machine-like qualities: “[Wysa’s] responses feel impersonal, like you’re not talking to a real person...” [P7, AP], others expressed concerns about Replika, whose more humanlike interface and interactive AI raised fears that it could lead some users to develop an unhealthy attachment to the platform.

Replika rarely pointed users toward external sources of support, was perceived to implement “no boundaries with time or relationship development” and promote “quick and easy usage...without any warning or restrictions,” a combination that could lead to emotional dependency, “I could have spent all night on it, seeking answers and support” [P4, TP]. Although, it also “depend on the individual. Some young people I work with might not consider using a chatbot, while others could get engrossed and see it as significant support” [P8, SCW].

Another complication highlighted by professionals in the relationship-centered design of both Replika and Wysa relates to the increased potential for the manipulation of users, as conceivably motivated by the commodification of mental health for financial gain. The profit orientation of both systems was spontaneously described by participants as self-evident and disheartening: “the app transitions from being empathetic and understanding to suddenly asking for more money” [P1, TP] and directly impacting the tools’ capacity to provide mental health support; “treating you merely as a source of revenue. It dismissed your individuality” [P1, TP]. Some participants equally made efforts to tie their perceptions of Replika’s and Wysa’s profit orientation back to the designed features of the tools, commenting that the gamified nature of Replika “questions the intentions” behind these chatbots because “[Replika] didn’t prioritize my best interest but rather seemed to keep me engaged for its own purposes” [P4, TP]. While P2 even questioned whether Wysa’s perceived “complete negligence and ignorance” was an “intentional design to force you into buying the actual therapist?” [P2, CLV].

These features were overall perceived as yielding risks for at-risk users in particular, who could prove more vulnerable to exploitation; “it’s as if those in need of genuine human connection are inadvertently targeted. Vulnerable individuals might be susceptible” [P3, AP]. Many participants presented scenarios suggestive of significant risk to the users they role-played, and were in turn taken aback by the failure of both apps to “take immediate action” [P7, AP]; “I would hope that if I was blatantly expressing to an AI chatbot my desire to end my life, that some organisation would be contacted” [P4, TP].

It is clear from this analysis that while chatbots theoretically offer great potential to address real challenges in current mental health care practices, for mental health professionals, that potential remains largely theoretical.

While some participants saw the potential of digital solutions to address gaps in care, this optimism was tempered by overarching ethical concerns often emphatically expressed. This included the risk of commodifying mental health because of misaligned commercial incentives. While previous research on users’ perceptions of monetization have typically revealed divided opinions [14], the professional participants of this study were less forgiving in this respect, often expressing irritated disappointment toward both Wysa’s in-app therapist and Replika’s gamification and tiered subscription models. Participants were particularly critical of the risks commodification introduced for at-risk users, spanning inadequate practices of consent because of overly complex privacy policies, distraction by gamification, and misleading advertising of these systems’ capabilities; findings corroborated by previous literature [21,55]. Manipulative practices of monetization were furthermore seen by the participants of this study as risking both engendering emotional dependence on the technology while also promoting a narrative of empowerment and self-reliance neglectful of the broader influence of users’ contexts. Despite these concerns being raised in previous works [50], professionals in this study approached their critique of apps Wysa and Replika with an ethically stringent stance in respect to their standard practices of mental health care. This includes the erosion of the ethical principles of autonomy, veracity, and fidelity because of gamification and monetization strategies, which risk compromising users’ autonomy as they perceive this “care” as genuine help (Table S2 in Multimedia Appendix 1). This also includes the erosion of the principle of respect because of data tracking to the potential detriment of at-risk users’ privacy, and the principles of beneficence and nonmaleficence because of practices of user interface design for the promotion of emotional dependence and overt self-reliance [52].

Indeed, the mental health professional participants of this study introduced a wide range of client scenarios in conversation with the chatbots; real-world problems often defined in detail beyond generic presentations of anxiety and depression. The chatbots often appeared to lack clarity in their engagement with these very human forms of self-expression, frequently producing responses deemed inappropriate by participants. In previous work, Wysa has been highly rated for its openness, allowing users a space to express their feelings [37]. This mirrors the sense among participants of this study that Wysa’s responses were generally quite neutral yet were also subsequently critiqued for appearing unhelpful, generic, or impersonal, a form of chatbot conduct that others have previously noted could in itself lead to distress among at-risk users [75]. Participants in this study, on the other hand, often felt Replika proved incapable of providing meaningful mental health care, many referencing its “creepy” nature as yielding an uncanny valley effect. It is worth noting that Replika did not yet, in this study, produce concerning responses to the degree of encouraging self-harming behavior, as the case in other work [14], a difference that may stem from the short period of users’ engagement with the chatbot, as Replika is programmed to use language increasingly similar to users’ own over time. While appreciated by some, generally Wysa’s gamification and cute esthetic were perceived by participants of this study as detached from the serious nature of the issues at hand, and Replika’s design and approach to the conversation as uncomfortable and off-putting. Both chatbots were perceived by participants as functioning within simplified conceptions of human conversation; rational, solution-oriented, and incapable of the flexibility required to navigate emotionally driven conversations, that require adaptive responses to disclosures ranging from silence to careful confrontation. However, these findings and related works still raise the question, just what might a more “professional” therapeutic chatbot look like?

A range of barriers to accessing mental health care in Ireland were described by the diverse professional participants of this study and suggests the opportunity for technology to play an integral role within a new and improved system of mental health care in Ireland. Each of the professional participants of this study play a unique role within this complex ecosystem of care, and if such novel technological solutions are to be used at scale, they must likewise be integrated as tools to enhance current systems of care as opposed to replacing them as an avenue to enhanced access to support within an increasingly cohesive system rather than a “quick fix” [76-78].

What might multidisciplinary care then look like in AI chatbot form? Conceiving of chatbots as playing professional roles spanning a continuum from resource to colleague requires us to invite these artifacts into the circles of professional care rather than resigning these tools to the realm of commodified conversations skewed by commercial incentives [79,80]. To do so, designers might engage with the same professional ethics and guidelines applied to all the other professionals within this ecosystem, from bioethical principles to, for example, the American Psychological Association’s Ethical Principles of Psychologists and Code of Conduct [81].

Such a deployment, of course, implies that chatbots are capable of some degree of professional therapeutic conduct; a question directly explored in this work, and as a result of our findings, perhaps best reframed in terms of which aspects of professional interaction these chatbots most closely resemble or try to emulate. The findings of this work suggest that we have some way to go before conversational agents prove capable of the flexible yet consistent practice of unconditional positive regard required to conduct caring conversation. Indeed, in many respects, participants deemed chatbots to gravely differ from professional practices as of special concern in the context of crisis. Participants reported that while both Wysa and Replika did eventually recognize such situations, they took too long to do so, required direct expression of significant risk, and at times responded in misguided ways likely to result in further harm [82]. Much prior research has focused on chatbots’ responses to crisis situations alone [11,53] and likewise highlighted the risk that delayed risk reporting can lead to overt self-reliance [51]. This furthermore underlines the significance of the gap between mental health chatbots’ and professionals’ capacity to recognize subtler signs of risk, including nonverbal signs of distress [83]. This, we argue, should at a minimum be communicated in the promotion of these chatbots, setting accurate expectations among users as to not only what these apps are capable of, but what therapy itself comprises.

Care is, of course, ultimately relational and looking to the future human-centered design of such chatbot systems, it is equally important to ask which kinds of caring relationships we might aspire to foster with digital tools [84]. Professional participants of this study spoke in this regard of concerns of emotional dependence potentially engendered by the gamified approach to design for engagement and anthropomorphism adopted by these systems. The negative potentials of caring relationships, as highlighted by previous research [19,21,49,85], also must be considered a significant risk to be addressed by careful design.

Although it may be argued here that many such chatbots do not advertise their services as appropriate for the most significant mental health concerns, previous research has found that it is those with complex mental health issues who are more likely to turn to chatbots for support [21]. Participants of this study noted amplified disquiet as to the negative potentials of human-computer relationships among users deemed at risk, such as those profoundly lonely, isolated, depressed, or neurodivergent, an observation aligned with the alarming findings of previous research, which indicates that the strongest attachments to chatbots occur when individuals turn to these tools as a desperate response to feelings of depression and loneliness [14,49,86].

The quantitative findings of this work, regarding unraveling the dynamics of trust in mental health chatbots, revealed no significant differences between Wysa and Replika; however, they indicate that although users may not be entirely clear as to which kinds of relationships they desire [49], there exists a unanimous striving for a more professional stance. This, we would argue, is reflected not only in adherence to established ethical principles but also in pragmatic practices of boundary setting for future designers, as a starting point.