According to the World Health Organization [1], the COVID-19 pandemic caused mental health services to collapse in 93% of countries worldwide, given the extreme demand [2] in the face of the fragility of lives, economic downturn, sociopolitical tension, and conflicts [3]. Such extreme shortage of services prompted researchers, governments, and service agencies to develop viable options that can promote mental health self-care in the general population, prevent the worsening of mental health challenges, and promote overall mental health well-being, so that populations as a whole can be more resilient in facing future challenges rather than relying solely on a reactive, remedial treatment model that focuses on alleviating distress but falls short on promoting public mental health [4].

In Hong Kong, the public health care system faces similar challenges, with marked delays for the great majority of people seeking mental health services for common mental health disorders. The public psychiatry outpatient clinics documented more than 52,426 new bookings in the year 2023, with the longest waiting time reaching a concerning 100 weeks for people with relatively stable mental health conditions [5]. This delay is partly attributed to the current service model that relies on the physicians’ referral for appointment-based psychiatric consultations, where priority is given to people with severe mental illnesses. This causes most service seekers to queue up for their first consultation, with hardly any bridging service offered while waiting. There are also very limited alternatives other than private practices, which may not be affordable for most people.

Under such circumstances, chatbot technology offers new opportunities to bridge the gap by promoting mental health self-care [6]. Chatbots are computer programs designed to engage users through visual, spoken, or written language [7,8]. They are highly accessible and scalable as they provide around the clock support at people’s convenience, overcoming the limitations of traditional appointment-based therapy that is often offered only during working hours [9]. Interacting with a chatbot also reduces social pressure when disclosing sensitive and emotional issues, providing users with a confidential and nonjudgmental environment [10-12]. While some might question the loss of human touch with these virtual agents, Park et al [13] demonstrated that chatbots created with humanlike characteristics and interaction styles can elicit cooperation and trust from users, boosting the effectiveness of mental health therapies. Bickmore et al [14] also showed that individuals could develop therapeutic bonds with chatbots just as well as with human therapists, and sometimes prefer them over humans [15].

Currently, the 2 main types of mental health chatbots documented in the literature are rule-based chatbots, which use predefined rules or decision trees, and generative artificial intelligence (Gen AI) chatbots, which use machine learning and natural language processing. Abd-Alrazaq et al [10] identified that 89% of the published studies on mental health chatbots used the rule-based approach due to its suitability for handling simple, straightforward, and well-structured tasks. The controllability of using predefined responses reduces the likelihood of unforeseeable errors or harmful responses and ensures the evidence base of responses.

Evidence on chatbot-based interventions for mental health has steadily grown, although the findings remain mixed. Early studies suggested that chatbots can serve as an effective medium for delivering mental health interventions. For instance, Fitzpatrick et al [16] compared a cognitive behavioral therapy–oriented chatbot (Woebot) with an information-only control group and found that participants using Woebot showed significantly greater reduction in depressive symptoms (Cohen d=0.44), whereas those in the control group did not. Inkster et al [17] examined the mobile chatbot app Wysa using real-world data and compared high-use and low-use users and also found that high-usage users experienced notably greater reductions in depressive symptoms (Cohen d=0.47).

A recent meta-analysis by Zhong et al [18], encompassing 18 trials, identified moderate reduction in depression (g=–0.26) and anxiety (g=–0.19) after 8 weeks of treatment, although these effects did not persist at the 3-month follow-up. Meanwhile, Li et al [19] found a larger effect size in reducing depressive symptoms (g=0.64) and distress (g=0.70), but these same interventions did not enhance psychological well-being (g=0.32). On top of these mixed findings, studies exploring chatbot interventions are often hampered by small sample sizes or nonrandomized designs. In addition, many interventions focus narrowly on symptom alleviation rather than broader prevention work.

Recognizing the prevention potential of chatbots requires a shift in focus from merely symptom management to cultivating personal assets that empower individuals to engage in self-care behaviors. Self-efficacy, as described by Bandura [20], is a key determinant of health behavior change. Research has shown that higher self-efficacy is associated with greater engagement in health-promoting behaviors [21,22]. Enhancing self-efficacy is also suggested to be an effective strategy in mental health promotion [23]. Within the context of recovery and rehabilitation, self-care self-efficacy, that is, one’s perceived ability to perform relevant self-care activities, including strategies on relaxation, visualization, positive reframing, maintaining a positive attitude, and enjoying life [24], has been shown to relate positively with better adjustment in various populations, including people with cancer [25], people who have experienced stroke [26], and people with renal disease receiving hemodialysis [27]. Mak et al [28] showed that self-care self-efficacy substantially mediates the effect between social support and psychological adjustment, highlighting the important role of self-care self-efficacy in one’s mental health.

Concurrently, mental health literacy is another facilitator of self-care behaviors [29]. Mental health literacy is the knowledge and beliefs about mental disorders that aid in their recognition, management, or prevention [30]. Mental health literacy enables the critical evaluation of incoming information, facilitating individuals to make informed choices and avoid misinformation [31]. Choi [32] demonstrated the linkage between literacy, self-efficacy, and health-promoting behaviors, such that self-efficacy increases when an individual becomes more capable of making informed choices. This enhanced efficacy reinforces the uptake of health-promoting practices. Studies have shown that mental health literacy is crucial in encouraging early intervention and effective self-management, ultimately leading to better mental health outcomes [33,34].

Using a randomized controlled trial design, this study evaluated the effectiveness of a chatbot-based intervention (chatbot group) compared with a waitlist control group in improving self-help self-efficacy, mental health literacy, self-care intention, and the actual uptake of self-care behaviors. We had 3 hypotheses:

The study is a 2-arm, assessor-blinded randomized controlled trial comparing the effectiveness of a chatbot-based intervention with a waitlist control group. This study was preregistered on ClinicalTrials.gov (NCT05694507).

The trial protocol was approved by the Survey and Behavioral Research Ethics Committee of the Chinese University of Hong Kong (SBRE-22-0383). Before enrollment, all participants gave their informed consent on a web-based platform (Qualtrics) with their digital signature. They were informed about the study information and their rights to opt out or withdraw at the time of their choice without any consequence. The data were anonymized so that participants could not be identified. A compensation of HKD 50 (US $6.50) in payment voucher was offered to the participants for each valid completion of the study questionnaire.

Participants included were (1) aged ≥18 years, (2) able to read and understand Chinese and spoken Cantonese, and (3) able to access the internet. As the contents of chatbots were extracted from an existing web-based mental well-being self-care platform, Jockey Club TourHeart+ Project [35], existing users of the platform and people who had participated in related research studies were excluded to avoid contamination. They were being batch-recruited through advertising on social media platforms (eg, Facebook and Instagram [Meta Platforms, Inc]), mass mailing at investigator’s institutions, and snowball sampling across 2 batches.

Upon registration, registrants gave informed consent on the web and were invited to complete a set of baseline questionnaires on a web-based surveying platform (Qualtrics; Qualtrics International Inc). The questionnaire was self-administered. A validation check, including validation items and time spent, was applied to confirm the quality of responses. Only those who gave valid responses to the baseline questionnaire were considered as valid participants in this study.

Participants were randomized at a 1:1 ratio (chatbot vs waitlist control) using a computer-generated sequence created independently by the platform’s computer engineer. The randomization algorithm was validated before the study launch by an external data manager, who confirmed the sequence was free from identifiable patterns. To maintain concealment, neither the principal investigators nor the research staff had access to the algorithm or allocation list, which was stored on a secure server with restricted permissions. The blinding of participants was not feasible due to the nature of the intervention, but the research team remained unaware of group assignments until the study concluded. All allocation logs were locked and periodically audited to ensure no tampering occurred. Data analysts who conducted the final analyses were masked to participant identities throughout.

Ten topic-specific chatbots covered a variety of topics, including relationships, stress management, value clarification, and emotion regulation, and used prevailing methods that have evidence in promoting mental health self-care and mental well-being, such as behavioral activation and positive psychology. In addition to psychoeducation about the topic, each chatbot also embedded related exercises, including mindfulness breathing and muscle relaxation practices. The materials in the chatbots were introduced by the virtual assistant called Boon, in Cantonese Chinese, the local dialect used in Hong Kong. It used person-centered language to make the conversations more humanlike with a personal touch. All contents and their rule-based responses were created by a team composed of a clinical psychology professor, a team of registered clinical psychologists, and psychological well-being officers, who are bachelor-level workers trained to provide low-intensity psychological interventions to individuals with common mental disorders. The 10 chatbot topics included the following:

Participants interacted with these chatbots using predesigned categorical choice options. Each time the user clicked a chatbot, it welcomed the user and started the conversation. The chat progressed in turn-taking fashion between the chatbot and the user and new response options were presented after each response was made by the chatbot. Screenshots of chatbots are presented in Figure 1. Each chatbot took approximately 10 to 15 minutes to complete the entire episode of conversation. Whenever premature termination of the chatbot was detected, it asked whether the user would like to continue the same topic that they had left unfinished when they logged in again.

Upon enrollment, participants registered their accounts free of charge using a valid email on a web-based platform that contains the chatbots. Participants in the chatbot group completed one topic-specific chatbot per day for 10 consecutive days in the experimental phase, followed by a 7-day free-access period to the chatbots they had previously completed. During the experimental phase (the first 10 days), the 10 chatbots were distributed to each participant in random order, accounting for practice effect due to the specific sequence. Participants were required to complete the chatbot within the same day of distribution. Access to the chatbot expired the day after distribution, and participants were unable to revisit the same chatbot thereafter. During the subsequent free-access period, participants could freely browse through the chatbots they had completed previously for revision. However, any chatbots that participants did not complete during the experimental phase were not accessible during the free-access period.

Concurrently, participants in the waitlist control group did not have access to the chatbots until the end of the study. Upon completion of the study, all participants, including the ones from the waitlist control group, were granted access to all chatbots migrated to an integrated digital mental health platform, Jockey Club TourHeart+ [35].

All participants were invited to complete 3 questionnaires: baseline before randomization, on day 11 (after the test), and day 21 (follow-up) after group allocation. A validation check was applied to these questionnaires to ensure the quality of responses. A valid response was defined as >80% correctness on the attentional check items and within 2 SDs of average completion time. Participants were encouraged to inform researchers of any discomfort or adverse events caused by the chatbot at any time during the study.

In the preregistered protocol, participants were randomly assigned to conditions with or without push notifications based on computer-generated random digits. However, during the actual trial run, a coding error prevented the algorithm from functioning as intended. Consequently, we decided to remove push notifications for all participants in the first batch. After observing a low completion rate of the chatbot tasks among participants in this initial batch, we modified the procedure for subsequent participants. Specifically, all participants recruited from the second batch received daily email reminders for 10 consecutive days following group allocation. These reminders served to notify participants about upcoming questionnaires, and for those in the chatbot group, each reminder included an additional link directing them to the chatbot.

This study adopted the Chinese version of all measurements. If there was no published Chinese version, standard translation, back-translation, and verification were conducted [36]. Information about participants, such as age, gender, education level, marital status, monthly income, occupational background, religion, experience in mental health–related services, and mindfulness practice, were included in the demographics section. Use data regarding the initiation and completion rates of chatbots were recorded in the backend system.

Given there was no previous study on chatbot intervention targeting self-care behaviors, the sample size was estimated based on a generic effect size of small effect (f=0.1). Calculated using G*Power (version 3.0), 222 participants (74 per group) were required to detect a small within-between interaction effect of f=0.1 on a repeated measure ANOVA (90% power, significance=0.05, 3 groups, 4 measurements, assuming correlation among repeated measures=0.5, ε=1). This number was inflated to a planned sample size of 278 (139 per group) to account for an estimated 25% follow-up attrition rate.

All analyses were performed using SPSS (version 29; IBM Corp) [50]. ANOVA and χ² analysis were used on demographic variables (eg, age, gender, education, and baseline mental health variables) to examine differences between groups. Primary outcomes of self-care self-efficacy (SUPPH), behavioral intention (eTAP), self-care behaviors (SCBI), and mental health literacy (Mental Health Literacy Scale) and secondary outcomes of depressive symptoms (PHQ-9), anxiety symptoms (GAD-7), and mental well-being (PERMA) were analyzed using linear mixed models (LMMs) with an intention-to-treat approach. Mixed models were a well-established method for imputation of missing values using restricted maximum likelihood [51].

In the LMM, random intercept models were fitted with intercepts included for each individual. The fixed effects included group (chatbot group or waitlist control group), time (baseline, after 10 days, and 1-month follow-up evaluation), and their associated interaction effects. Since receiving push notifications has a significant effect on adherence, it was included as a covariate in the LMM analyses. To follow up on any significant interaction effect (2-way time × group interaction), post hoc pairwise comparisons with Bonferroni adjustment were conducted by looking into the effects fixing at each level of time and condition. Pre-post and pre–follow-up repeated measures effect sizes (Cohen dRM) were calculated following the study by Du et al [44]. Between-group effects and interaction effect sizes were calculated using a web-based calculator, Psychometrica—form 2 and 3, respectively [52]. Effect sizes were interpreted according to Cohen guidelines, where 0.2 indicates a small effect, 0.5 a medium effect, and 0.8 a strong effect.

As for sensitivity testing, another series of per-protocol analyses were conducted using the same procedure as mentioned earlier. Per-protocol criteria were defined as the completion of all assessments without missing data and completing at least 1 chatbot for those who were assigned to the chatbot group.

Participants were recruited between January 2023 and March 2023. A total of 468 individuals registered for the study across 2 batches, gave informed consent, and completed the screening survey to check for their eligibility. In total, 33.8% (158/468) of the participants did not meet the inclusion criteria, and 4.9% (23/468) of them provided invalid responses on the baseline survey and were excluded. In total, 0.4% (2/468) of participants withdrew afterward before the commencement of intervention due to personal reasons and were further excluded. The final intention-to-treat sample consisted of 285 individuals with 140 individuals being randomized to the chatbot group and 145 in the waitlist control group. Figure 2 provides the CONSORT (Consolidated Standards of Reporting Trials) flowchart, while the CONSORT-EHEALTH checklist is included as Multimedia Appendix 1. Results of t tests (2-tailed) and χ² tests showed no significant differences in the demographics and outcomes between the chatbot and control groups and between those who received and did not receive notifications. Table 1 summarizes the demographic characteristics of the participants.

Among the 285 participants, 35 (12.3%) of them dropped out after the assessment, and 51 (17.9%) dropped out at 1-month follow-up. Significantly more participants dropped out in the chatbot group than in the control group at follow-up (χ²₁=16, P<.001). Their reasons for withdrawal were unknown since we did not administer an exit survey. No instance of serious adverse events was reported throughout the study period. Notification status (received or not received) had no significant effect on attrition after the evaluation (χ²₁=0.1, P=.71) and at follow-up (χ²₁=3.3, P=.07).

Mean baseline depressive and anxiety symptoms were significantly higher among those who dropped out after the test (PHQ-9: mean 9.17, SD 5.16; GAD-7: mean 10.14, SD 5.70) than those who stayed (PHQ-9: mean 7.25, SD 5.09; GAD-7: mean 7.73, SD 6.05; PHQ-9: t₂₈₃=2.09, P=.04; GAD-7: t₂₈₃=2.23, P=.03). Their overall mental well-being (mean 5.46, SD 1.55), as assessed by the PERMA profiler, was also marginally lower than that of the retained participants (mean 6.11, SD 1.45; t_42.70=–2.35; P=.01).

Similarly, those who dropped out at follow-up had significantly higher PHQ-9 scores (mean 9.57, SD 5.37) and higher GAD-7 scores (mean 11.22, SD 6.86) than those who stayed (PHQ-9: mean 7.03, SD 4.97; GAD-7: mean 7.33, SD 5.64; PHQ-9: t₂₈₃=3.26, P=.001; GAD-7: t₂₈₃=4.28, P<.001). These revealed that selective bias was presented, with retained participants having better initial mental health in terms of lower levels of depression and anxiety symptoms and higher levels of mental well-being.

All the 140 participants in the chatbot group initiated at least one of the 10 topic-based chatbots during the first 10 days. 12 participants (8.6%) terminated the first chat prematurely and did not start another. Of 140, 128 (91.4%) finished at least one entire conversation. In total, 33.6% (47/140) of participants completed all 10 chatbots, and 14.3% (20/140) of them completed 9 chatbots. The remaining percentages were evenly distributed across the number of chatbots completed. On average, participants completed 6.78 chatbots (SD 3.50). Push notifications had a significant effect on the chatbot completion rate (t₁₃₈=–4.91, P<.001). The 85 participants who received push notifications completed significantly more chatbots (mean 7.86, SD 3.30) than the 55 participants who did not receive notifications (mean 5.11, SD 3.14).

Figure 4 shows the changes in the secondary outcomes overtime by condition graphically.

Results of the per-protocol analysis revealed the same pattern of results on primary outcomes as the primary analysis. Slight discrepancies were observed in the secondary outcomes of overall well-being. The time × group interactions were significant in the sensitivity test (F_2,295.27=3.09, P=.047). Following up on this significant interaction, the pattern was identical to that of positive emotions noted earlier. Improvement from baseline to 10 days was only significant in the chatbot group (mean difference 0.25, 95% CI 0.04-0.46; P=.02) not in the control group (P>.99). Furthermore, later both groups improved significantly at 1 month (chatbot: mean difference 0.71, 95% CI 0.46-0.95, P<.001; control: mean difference 0.54, 95% CI 0.33-0.76, P<.001). Table 6 outlines the result table comparing intention-to-treat and per-protocol analysis.

This study evaluated the effectiveness of topic-based chatbots on self-care efficacy, the intention and actual uptake of self-care behaviors, and mental health literacy, alongside associated mental health outcomes, compared to a waitlist control condition. Our findings align with previous research showing immediate improvements, with small-to-medium effect sizes, in mental health outcomes [16-19]. Our primary hypothesis was partially supported as the chatbots did not perform better than the waitlist control on users’ self-care efficacy but significantly promoted their intention and mental health literacy in the short term.

We attribute these findings to the nature of the guidance provided by the chatbots. The chatbots excel at delivering clear information surrounding mental health and actionable steps that users can follow immediately, such as mindful breathing and relaxation exercises. These steps were straightforward and easy to implement, requiring minimal effort. The information presented quickly boosted participants’ literacy. The guidance on short exercises enhances users’ willingness to try out these self-care actions and thus increases their behavioral intention in the short term.

However, enhancing self-efficacy involves more than just doing the practices; it requires building confidence through personalized, reflective, and emotionally supportive interactions [18]. Rather than allowing free-text entry, our rule-based chatbots were developed with predefined, hardcoded responses. While this approach provides structured and predictable interactions, it lacks the flexibility and responsiveness required to address the dynamic nature of mental health concerns and provide personalized reinforcement. The preset multiple-choice type of options also limits users’ depth of reflectivity of their practice experiences necessary for developing self-efficacy. The short 7-day duration of the chatbot intervention may also be insufficient for users to internalize new skills and gain confidence in managing their mental health independently.

The nonsignificant findings regarding the secondary mental health outcomes were disappointing. However, this is understandable as we did not restrict control participants from using other means to take care of their mental health during the study period. In this digital era, people have all sorts of channels to access mental health promotion tools, be it apps, websites, or even joining other intervention studies. This may explain why we observed a sharp improvement among our control participants at the 1-month mark as they may have started exploring other alternatives. We regret for not administrating an exit survey to confirm this claim, future trials should include this to gain a better understanding of attrition.

Among the few studies that discussed the chatbot’s effect on well-being, minimally small differences between the intervention and waitlist groups were obtained [53,54]. Similarly, we found small differences between the groups on most well-being domains. The chatbots were more able to enhance users’ positive emotions. We suspected the chatbot’s engaging features provided satisfying experiences and positive reinforcement that are effective toward their mental health, at least in the short term.

We acknowledge the rapid advancements in chatbot technology; at the time when this study was planned, Gen AI chatbots powered by large language models (eg, OpenAI’s ChatGPT, Microsoft’s Azure) were not as accessible as they are nowadays. Our chatbots were designed with decision trees following the convention back then. Previous research on rule-based chatbots reported negative experiences and perceptions, such as being repetitive, boring, and unfunctional, making users feel bothered [17,55,56]. These issues may also apply to this study. Unlike rigid rule-based scripts, Gen AI chatbots learn from extensive datasets, refining responses in real-time. They can interpret users’ inputs, adapt to particular context, and deliver personalized content to improve user engagement. However, Gen AI chatbots are not free from challenges. They could generate responses that seem convincing but are factually incorrect, that is, “hallucinations” [57]. While these models hold the potential to expand mental health support and reach underserved populations, the need for transparent design, rigorous evaluation, and appropriate clinical safeguards cannot be understated [58].

In addition, this study recruited adults aged ≥18 years who possessed proficiency in using computers and comprehension of language. Recruitment was done primarily through web-based platforms and snowballing, contributing to a selective sample of individuals with better eHealth literacy, greater interest in mental well-being, and a higher acceptance to engage with mental health chatbots. This aligns with our attrition analysis, which found that people who stayed had better baseline mental health. Therefore these results cannot be generalized to the broader population of individuals who might not be as technology-savvy and mentally well. Future studies could investigate the feasibility of these chatbots in other populations, such as older adults, adolescents, and people in rural areas.

Despite these limitations, this study is one of the few studies that tested the effectiveness of chatbots on self-care intentions and behaviors, using a rigorous, adequately powered randomized controlled trial design. There is an abundance of mental health self-care tools available in the market, offering a wide array of options for consumers. While this proliferation of resources broadens the choices for individuals seeking mental support, it also creates risks. Many of these tools are not empirically tested or are misinformed [59-61]. The topic-based chatbots used in this study were designed with evidence-based approaches (eg, cognitive behavioral therapy and mindfulness-based intervention), and the composition of the content was developed and supervised by clinical psychologists. There was no adverse event reported during the trial. In this sense, our chatbots are empirically supported and safe to use.

In conclusion, this study provides empirical support for the preventive potential of rule-based chatbots in enhancing mental health literacy and increasing the immediate intention to practice self-care. However, their ability to promote self-care self-efficacy and long-term mental health improvements is limited. The chatbots’ structured, evidence-based content likely contributed to the improvements in mental health literacy and short-term behavioral intentions, while the lack of personalization may explain the limited impact on self-efficacy. These findings highlight the need for more advanced and personalized chatbot interventions to leverage technology effectively for mental health promotion. Nonetheless, these topic-based mental health chatbots may help bridge the gap between the high demand for mental health services and scarce resources by initiating people’s self-care efforts. Future research should continue to empirically test Gen AI chatbots, explore the specific mechanisms through which chatbots cultivate personal assets, examine their effectiveness in diverse populations, and explore hybrid approaches combining rule-based and Gen AI systems.