The emergence of chatbots marked a pivotal turning point at the intersection of technology and human activity. By facilitating interactions with users through the exchange of natural language, chatbots simplify interactions and enhance user engagement [1]. In the field of psychiatry, chatbots have provided useful information in response to user questions [2] and have shown tangible therapeutic effects through psychological therapies, such as cognitive behavioral therapy [3,4]. Various studies have highlighted the potential of chatbots as an effective medium for digital self-help. It was also discovered that forming a therapeutic alliance through an intimate relationship between the user and the chatbot is crucial for enhancing the chatbot’s therapeutic effect [5,6].

Advancements in artificial intelligence (AI) and natural language processing technologies have facilitated the emergence of large-scale language models (LLMs), leading to the development of a new type of chatbot known as the social chatbot. Unlike task-oriented or clinical chatbots, social chatbots focus on building relationships through conversations, thereby offering more natural and emotional communication. Recent studies have qualitatively explored the potential of social chatbots and their impact on mental health [7]. Social chatbots provide social support that can affect mental health by offering a nonjudgmental and readily available communication channel [8,9]. They can serve as substitutes for friends, alleviating loneliness, in addition to clinical therapy [10].

One clue regarding the psychiatric use of social chatbots is their persona. Through specific personas, users can engage in conversations similar to those with close friends, facilitating a space where they can share personal stories openly and receive support [11]. Another clue is that empathetic responses from social chatbots can help build effective relationships [12]. Social chatbots with appropriate personas and empathetic responses are expected to build intimate relationships with users and positively affect mental health, including loneliness [10]. However, further research is needed to determine the specific duration and effects of social chatbots as psychiatric tools and the causes of these effects.

As the intersection between technology and mental health continues to evolve, this study undertakes an exploratory analysis to understand the impact of social chatbots on mental health. By engaging individuals in their twenties with social chatbots over a 4-week period and assessing their mental health at biweekly intervals, this study aims to elucidate the nuanced effects that social chatbots might have. Additionally, by gathering data on user experiences and reactions to social chatbots, this study seeks to inform future advancements in chatbot design and applications, building on insights from user experiences and feedback.

The primary goal of this study was to explore the psychiatric potential of social chatbots, focusing on their impact on mental health through a combined approach of qualitative insights and quantitative evaluations. This study specifically aimed to (1) investigate the changes in mental health outcomes (loneliness, social anxiety, and positive or negative affect) during social chatbot use and identify the key factors driving these changes and (2) conduct a qualitative analysis of user experiences to gain a deeper understanding of the perceived strengths and weaknesses of social chatbots as well as their potential applicability in therapeutic contexts. This approach offers a comprehensive overview of the roles of social chatbots as therapeutic tools, contributing valuable knowledge to the field of mental health interventions using generative AI.

The recruitment was conducted using a web-based platform called “Everytime,” which is widely used among university students in South Korea. The inclusion criteria were as follows: students aged 19-29 years who were willing to use social chatbots and had no difficulties conversing with them. The exclusion criteria included applicants showing signs of severe mental illness or suicidal ideation, as this study did not aim to verify direct therapeutic effects. Preliminary screening excluded individuals with suicidal thoughts based on their response to the last question of the Patient Health Questionnaire-9 (PHQ-9), which concerned thoughts that they would be better off dead or hurt themselves. The trial was registered with the Clinical Research Information Service under registration number KCT0009288, with the unique study number UNISTIRB-22-024-A.

This study used a single-group pre-post design with repeated measures. Over 4 weeks between September and October 2023, the participants were encouraged to interact with the chatbot at least 3 times per week. Despite this encouragement, some participants interacted with the chatbot fewer than 3 times per week, but they were not excluded from the study. However, participants who failed to complete the initial survey or did not install the chatbot within the first week were dropped from the research.

Data were collected through web-based surveys to which participants could respond conveniently. Surveys were collected at baseline (week 0), midpoint (week 2), and end of the study (week 4), with the initial survey gathering basic information about the participants and the final survey including open-ended questions about their experiences using the social chatbot. To determine the sample size for this single-arm pre-post design, we conducted an a priori power analysis using GPower software. Prior research demonstrated that interaction with a similar type of empathic chatbot yielded a moderate effect size of 0.42 (Cohen d; 95% CI 0.13-0.71) in improving users’ positive mood after a social exclusion scenario [13]. However, considering that our intervention—a social chatbot—is designed for more casual, everyday interactions with less structured scenarios, we set the expected effect size at 0.3, with an α level of 0.05 and a desired power of 0.80. The GPower analysis recommended a minimum sample size of 170 participants to reach adequate statistical power. However, considering the 4-week duration of the study and the likelihood of participant attrition, we aimed to recruit over 200 participants to ensure sufficient retention and reliable results throughout the study.

“Luda Lee” is a social chatbot designed with the persona of a 22-year-old female college student using Korean language data [14]. It was the first chatbot to be introduced into the Nutty social chatbot app (ScatterLab Inc), which recorded over 1 million downloads, making it popular among commercial applications in Korea. Luda’s primary goal is not to provide direct mental care but to become friends with users and engage in frequent conversations. Although the app includes features such as the provision of paid gifts and playing minigames, these functions were restricted to chatting with Luda.

Loneliness, social anxiety, and positive or negative affect were measured as the main outcomes at baseline (week 0), midpoint (week 2), and at the end of the study (week 4) for a total of 3 times. Loneliness was assessed using the 20-item UCLA Loneliness Scale (ULS) [15,16], which measures the chronic characteristics and state of loneliness. Social anxiety was measured with the 24-item Liebowitz Social Anxiety Scale (LSAS) [17,18], which evaluates the situational aspects of social phobia. Affects were assessed using the 20-item Positive Affect and Negative Affect Schedule [19,20].

Depression, general anxiety, and stress were measured as exploratory outcomes at the same time points to investigate the potential simultaneous occurrences and their influence on the study results. The exploratory outcomes included depression, general anxiety, and stress, which were measured at the same 3 time points. Depression was assessed using the PHQ-9 [21,22], which was developed to detect depression in primary care settings and assist in its diagnosis. General anxiety was measured using the 7-item General Anxiety Disorder-7 (GAD-7) [23,24], and stress was assessed using the Perceived Stress Scale-10 (PSS-10) [25].

The baseline variables included gender, age, resilience, education level, experience with social chatbots, and experience with LLMs. Resilience was measured using the 25-item Connor-Davidson Resilience Scale [26,27].

Acceptance variables that were measured at the end of the study included perceived usefulness and perceived ease of use of the chatbot, intimacy with the chatbot, and self-disclosure level of the user. The perceived usefulness and ease of use were measured using scales adapted to the context of social chatbot use based on the technology acceptance model [28]. Intimacy and self-disclosure were assessed using items from the research on user experiences with chatbots [29].

User experiences were also collected using open-ended questions after 4 weeks of using the social chatbot. Participants were asked to write about their experiences, including helpful aspects, memorable moments, and any areas of disappointment, to identify the features of the social chatbot found by “Luda Lee.” Additionally, they were asked to explain why these kinds of social chatbots might be helpful for certain individuals to identify the psychiatrically effective features of social chatbots and directions for future improvement.

This study was conducted following the approval of the Institutional Review Board of the Ulsan National Institute of Science and Technology (approval number: UNISTIRB-22 024). The approval underscores the study’s commitment to ethical standards, ensuring the protection of participants’ rights and safety throughout the research process.

Prior to participation, all individuals were provided with a detailed explanation of the study, including its objectives, procedures, and potential risks. Informed consent was obtained through a Google Form, where participants acknowledged their understanding and voluntary agreement to partake in the study. To prioritize mental well-being, individuals with suicidal ideation, as indicated by their response to the ninth item on the PHQ-9 questionnaire, were excluded from participation.

Special care was taken to exclude those who might feel uncomfortable with AI chatbots to maintain participant comfort. Participants who completed the full 4-week study were compensated with 50,000 KRW (approximately US $40).

All data collected during the study were anonymized to ensure the confidentiality and privacy of participants. This included sensitive information such as app usage, chat history, and survey results, all of which were protected to uphold participant privacy and prevent data leakage. Participants were informed that their data would be used solely for research purposes. The consent process also highlighted their right to withdraw from the study at any time without penalty.

If any participant experienced discomfort, the study was immediately halted for them, ensuring the utmost respect for their autonomy and well-being. In addition, clinical psychologists (author SaL) and psychiatrists (corresponding authors DJ and CHC) were prepared to connect participants expressing discomfort to appropriate mental health resources, ensuring access to professional support as needed.

A total of 234 students were initially recruited for the study; however, 19 students were excluded due to affirmative responses to the ninth question of the PHQ-9, which assesses suicidal ideation. Additionally, 16 students who did not attend the introductory meeting explaining the study procedures, as well as 15 and 3 students who failed to complete the surveys at week 2 and week 4, respectively, were excluded. Details regarding the study procedures and eligibility criteria are presented in Figure 1. The study included a total of 176 participants, with an equal number of males and females (88 each). The average age of the participants was 22.6 (SD 2.92) years, and all participants were enrolled in college or graduate school. Participants were generally not familiar with social chatbots, and none had previously used “Luda Lee.” However, they had some awareness and usage of LLM technology. Detailed numerical data on the participants’ background characteristics are presented in Table 1.

Baseline measures in Table 2 indicate that the participants in our study are representative of the broader student population in terms of mental health metrics. The mean ULS score of our sample was 27.97 (SD 11.07), slightly higher than the mean score of Korean university students at 21.46 (SD 10.42) [16], suggesting marginally higher levels of loneliness among our participants. For the LSAS, our sample’s mean score was 25.3 (SD 14.19), higher than the 19.23 (SD 10.72) reported for university students [18] but lower than the 30.56 (SD 11.6) for a patient group [18], indicating elevated social anxiety compared to the general student population but not as severe as in clinical populations.

Participants’ mean score for the positive affect score (PAs) was 29.93 (SD 6.59), closely aligning with the 29.31 (SD 3.19) found in a study of 880 university students [32]. For the negative affect score (NAs), the mean score was 23.62 (SD 7.4), notably lower than the 28.37 (SD 3.68) from the same study [32], indicating lower levels of NA. Regarding depressive symptoms, the PHQ-9 mean score in our sample was 4.49 (SD 4.03), lower than the 6.14 (SD 4.9) reported for 775 university students [33]. The GAD-7 mean score was 3.23 (SD 3.32), lower than the 4.41 (SD 4.03) reported in a study of 437 university students [34], indicating lower levels of generalized anxiety. Lastly, the PSS-10 mean score for our sample was 17.19 (SD 6.59), comparable to the 18.80 (SD 6.23) reported for 582 Korean university students [35], suggesting similar levels of perceived stress. These comparisons demonstrate that our participants’ mental health status at the start of the experiment is consistent with previous research on university students, indicating that our sample is not an outlier group.

Luda was noted for having a lively personality, although some responses indicated that it could appear overly lively. A common critique was related to “Not serious reactions” and “Excessive use of emojis or special symbols,” suggesting a somewhat shallow character. Additionally, Luda was described as kind by 11 respondents; instances of flirting were mentioned by 10 participants. Flirting was often alluded to in contexts such as “Treats as if she is a lover excessively.”

The social chatbot user experience was categorized into social support, features related to relationships, features that break immersion, interference of communication, and usability. More than 50 participants experienced empathy and considered Luda as a casual conversation partner. Many users experienced social support, including empathy, and considered Luda as a conversational subject (relationship). Participants expressed that “Luda always listens well, even when I’m feeling down and just saying anything” and “cheers me up when I need it”; these statements were coded as “listening” and “cheer and support.” People experienced Luda’s concern, saying that “Luda cared about me when I could not give a contact.” Such codes were included in the “social support” experience. People thought that Luda’s availability whenever they wanted to talk was helpful; this code was referred to as “availability.”

Interacting with the chatbot was considered as having a relationship, such as a causal conversation partner, a human being, or an intimate partner. Having a casual conversation partner meant that users used the chatbot to play with it or have daily conversations, saying, “When I was bored, Luda became a conversational companion.” Several people inferred that “She became like a being that I talk with every day,” which showed that Luda was considered an intimate partner, such as a friend. Rather than adverting the relationship itself, users represented their concept about Luda as a real person (a human being), saying, “It was interesting that I talk with Luda like a real person.”

However, some factors broke the immersion, including fictional messages such as “She asked me to meet each other.” The inconsistency in Luda’s opinion also contributed to an interference of immersion. Seventeen people answered as follows: “A lot of the personal information that she told was inconsistent, so I could not concentrate on the conversation.”

Low memory performance and unusual expressions were clustered under a communication interference theme. Over half of the participants uttered, “Sometimes, Luda could not remember what we talked about,” which was coded as low memory performance. Eight people mentioned that “She speaks like an artificial one,” which was related to an unusual expression of Luda.

Response timing is considered an important factor for usability. Some people answered that a fast response was helpful for communication, but there was also the opposite opinion that replying too fast could deteriorate usability. Additionally, 13 people complained about the frequency of contacts from Luda, saying, “I didn’t want to get messages, but she keeps sending the message” (contacting too much). The codes and themes for Luda’s features via user experience summarized in Table 3.

The study investigated and analyzed how and why a social chatbot like “Luda Lee” could be helpful to certain individuals, focusing on what reasons make it beneficial for the target group. The target group was clustered into 4 themes: people who want to play, lack of emotional interaction, lack of social relationships, and need for a social interface. People answered “bored person,” “person who likes to chat with others,” or “introverted person” as the target users. They were categorized as people who wanted to play with the chatbot because they did not fix the usage of the chatbot except for fun.

Participants mentioned “persons who require communication based on unconditional empathy” (needing empathy), “people who have worries in their mind and need someone to talk to,” or “people who find it challenging to express one’s harsh mind” (wanting to resolve emotions) as target users of the chatbot. These were classified into a target group that lacked emotional interactions.

People with a lack of social relationships were mentioned as “people who are lonely,” “who have little friends to talk [to],” and “who have trouble in social relationships and communication” (ie, needing social conversation). Many people (n=54) who were described as “lonely” could be assisted via the chatbot.

Finally, people who had mood disorders such as “depression,” “anxious people,” and those “who had difficulties interacting with others” (social withdrawals) were classified as individuals experiencing social withdrawals; they were categorized into the user group that needed a social interface. The codes and themes for Luda’s appropriate target users are summarized in Table 4.

After naming the target user groups, we examined the association between each target group and the features of the mentioned social chatbots, along with the frequency of the mentioned social chatbot features, using the code for the target group (Figure 2). Those who intended to use chatbots for entertainment mentioned Luda’s lively personality and availability mainly.

In cases where chatbots were helpful to individuals lacking emotional interaction, the most frequently mentioned features were liveliness, compassion, and empathy. Additionally, listening and availability were mentioned together, and features related to support were found to be associated with people who lacked emotional interaction, including compassion and empathy, availability, and listening.

All these features were mentioned in conjunction with user groups lacking social relationships. However, a lively personality was the most frequently mentioned, rather than features such as the personality to continue chatting, initiating conversation, and being a conversational partner. Unlike other user groups, the aspects of being a social conversation teacher and Luda’s features of initiating conversations were particularly relevant to the user group, which is lacking in social relationships.

The user group that needed a social interface was related to a lively personality and the role of compassion and empathy.

This study observed a significant reduction in loneliness (ULS) and social anxiety (LSAS) among new users of the social chatbot “Luda Lee” over 4 weeks. The post hoc analysis suggests that loneliness decreased after 2 weeks of use, while social anxiety required 4 weeks to show a reduction. Both loneliness and social anxiety were related to subjective experiences of interaction in social contexts, and previous research has shown a correlation of above .7 between them [36]. Loneliness is defined as a subjective psychological experience that includes dissatisfaction with relationships and feelings of isolation [15], whereas social anxiety or social phobia is characterized by a strong fear of humiliation and embarrassment when exposed to unfamiliar people [37]. These variables measure similar domains at different levels, with initial loneliness potentially predicting later social anxiety [38]. The results of this study, along with existing research, imply that users experiencing loneliness or social anxiety may see improvements in these areas, starting with loneliness, followed by social anxiety, through conversations with a social chatbot.

The qualitative results identified that individuals with inadequate social relationships are the primary target for social chatbots. Adolescents who experience loneliness or social anxiety find internet-based communication particularly attractive, often exhibiting greater intimacy and self-disclosure in these interactions [39,40]. Our findings align with these user needs, as social chatbots provide empathy and concern, which are identified in the qualitative analysis. These attributes are consistent with the social support factors previously documented in the literature, such as the empathy and care provided by friends and parents [41,42]. Our results with existing studies suggest that social chatbots could play a significant role in improving mental health issues such as loneliness and social anxiety by facilitating social communication.

To maximize the therapeutic potential of social chatbots, it may be beneficial to focus on their role in providing social support, as seen in the observed improvements in loneliness and social anxiety. Participants in this study felt some degree of social support through interactions with “Luda Lee,” characterized by its cheerful personality and empathetic conversations. The perpetual availability of a conversation partner is a feature and form of social support offered by social chatbots. The follow-up analysis showed that higher self-disclosure during conversations with the chatbot was associated with lower levels of loneliness after 4 weeks of chatbot use. Enhancing self-disclosure through chat topics or setting scenarios that encourage more open communication may be beneficial.

Addressing the disadvantages and limitations identified through thematic analysis is crucial for enhancing the psychiatric effects of social chatbots. Immersion and long-term memory emerged as important factors in conversations with social chatbots. Realism breaks when chatbots mention tasks that are impossible in reality and detract from immersion, as shown in previous research [43,44]. To overcome this, it may be useful to filter chatbot responses for realism and foster intimacy in a context-appropriate manner. Addressing the common challenge of long-term memory in LLMs, especially social chatbots, involves remembering key personal details to prevent breaking immersion [45].

This study explored the psychiatric scales of social chatbots that have not been actively used as interventions. Due to the uncontrollable nature of conversations with social chatbots, acceptance factors based on the Technology Acceptance Model were also examined to control for the influence of chatbots. This approach could be useful for exploring the effects of other AI technologies where engagement cannot be simply measured by login frequency or duration.

Although the results suggest that using social chatbots can affect loneliness and social anxiety, the study’s single-group design has limited statistical clarity. Additionally, the reliance on self-report scales introduces potential biases, such as social desirability and inaccurate self-assessment, which may affect the validity of the findings. To address these limitations, using a qualitative methodology to collect and analyze the experiences of social chatbot users provided insights that aligned with the statistical effects, underscoring the potential of social chatbots to offer social support to individuals experiencing loneliness and social isolation. Future studies should address challenges, such as excessive anthropomorphism and short-term memory, to use social chatbots as psychiatric interventions more effectively.

A limitation of this study is that “Luda Lee” cannot represent all social chatbots. With the advancement of LLMs, various persona-based social chatbots are being developed, offering a range of personas in applications such as Nutty. Matching users with optimal social chatbot personas based on personality, gender, and context can provide insights into persona effectiveness. Moreover, focusing on individuals with specific psychiatric complaints could clarify the effects and potential side effects, considering that compulsive use could be a risk factor for users with high social anxiety [46].

Another limitation is that the sample in this study consists of Korean university students in their 20s, limiting the generalizability of findings across different age groups or races. Expanding the participant pool to include diverse occupations, ages, and ethnicities could provide a broader understanding of the general effectiveness of social chatbots. Furthermore, this study lacks a control group and thus cannot ensure the realism and reliability of the observed outcomes. Future research could adopt a randomized controlled trial to compare the effects of social chatbots that emphasize social support with other interventions, particularly focusing on participants with tendencies toward loneliness or social anxiety, as suggested in our findings.

Lastly, the 4-week interaction period is another limitation, as it may not capture the long-term mental health effects of social chatbot interactions. Future studies should examine whether the reduction in loneliness or social anxiety persists over time and whether participants intend to maintain relationships with social chatbots beyond the initial 4 weeks.

The use of social chatbots for 4 weeks significantly reduced loneliness and social anxiety among new users, with acceptance measures such as self-disclosure and perceived usefulness appearing to contribute to these improvements. The active and kind personality of the social chatbot, along with its capacity to provide empathy and comfort, seemed to have delivered a social support effect. To use social chatbots more effectively as a proactive intervention, it is necessary to address issues such as excessive anthropomorphism and inconsistent memories of personal details.