To promote mental health for everyone, everywhere, the World Health Organization’s most recent global mental health report strives to motivate and guide revolutionary action [1,2]. According to estimates by the World Health Organization, in the first year of the COVID-19 pandemic, the incidence of both depression and anxiety disorders increased by more than 25% [3]. However, mental health services are few and still underutilized globally [4-6]. This was because traditional face-to-face mental health care still had many limitations, such as expensive treatment, lack of experienced therapists, poor service quality, geographical constraints, and delayed treatment [7,8], and the resulting stigma and discrimination are also considered the most important barriers to providing mental health services [9,10].

Because of their increased acceptance and accessibility, digital mental health interventions have emerged as an important research area with evidence-based psychotherapies implemented on digital platforms [11]. Conversational agent interventions (CAIs) [12] were the new wave of digital mental health interventions to cope with the insufficient and inadequate mental health services [1,13]. Substantial human health resources will be saved if the CAIs are proven effective and suitable for widespread implementation [14].

Software programs known as conversational agents (CAs) use artificial intelligence techniques to simulate human behavior and offer a task-oriented framework with evolving dialogue able to engage in conversation [15]. CAs are equipped with computer models that range from succinct decision trees, where different responses to a questionnaire result in different responses from the CAs to machine learning–based and natural language processing–based algorithms that classify real-time multimodal input into a user’s emotional state, allowing the CAs to respond empathically [16,17]. One of the most crucial features of CAs in mental health was interactivity, which was designed to promote a conversational process instead of a single psychological education, in which inputs and outputs are generated in unrestricted natural language rather than predefined or preprogrammed choices or messages [12]. The other was automation, which means that most CAs for mental health issues can independently provide automated services to users without the participation and guidance of human [18].

Certain studies claimed that CAIs can help users feel accompanied and understood [19,20]; several studies found that users establish therapeutic bonds with chatbots [21-24]; and other studies revealed some hazards associated with CAIs, including misunderstanding that may result in inefficient or even harmful interventions, a lack of crisis warning systems, and a lack of privacy protection [25]. An increasing number of CAIs have emerged in recent years, as the digital medical and mobile health fields have grown [26], solving and improving a larger range of mental health issues in addition to depression and anxiety. Therefore, a thorough systematic review and meta-analysis of CAIs for mental health problems are urgently needed.

In this study, we outlined the clinical and nonclinical features of CAIs in mental health using a systematic review. We then evaluated the short- and long-term efficacy of CAIs for different mental health outcomes via a meta-analysis and assessed whether various characteristics related to the intervention and sample moderated the observed effect sizes.

We have reported the findings in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses; Table S1 in Multimedia Appendix 1 [27-58]) guidelines [59]. The study protocol was registered in the PROSPERO database (CRD42022350130).

As mentioned in the initial registration, we searched 6 major web-based databases (Embase, MEDLINE, PsycINFO, CINAHL, Web of Science, and Cochrane), dated from the establishment of the database to October 30, 2021, and updated to May 1, 2022, using the following search terms: (“conversational agent*” OR “conversational system*” OR “dialog system*” OR “dialogue system*” OR “assistance technolog*” OR “relational agent*” OR chatbot* OR “automat*” OR “virtual human*” OR “virtual agent” OR “virtual coach” OR “virtual therap*” OR avatar OR “artificial Intelligence”) AND (“depress*” OR “anxiety” OR “agoraphobia” OR “phobia*” OR “panic” OR “mental health” OR “mental illness*” OR “mental disorder” OR psycholog* OR “affective disorder*” OR “bipolar” OR “mood disorder*” OR “psychosis” OR “psychotic” OR “schizophre*” OR “well-being” OR “well-being” OR “quality of life” OR “self-harm” or “self-injury” OR “stress*” OR “distress*” OR “mood” OR “loneliness” OR “social isolation” OR “autism” OR “suicid*” OR “cogniti*” OR insomnia OR emotion* OR affect*) AND (“randomized trial*” OR “controlled trial*” OR “randomised trial*” OR RCT OR RCTs). In addition, the reference lists of the included original studies and previous reviews were manually searched to identify any further eligible studies. To reduce the possibility of publication bias, eligible published and unpublished papers were searched for inclusion.

The following data were extracted: (1) basic study characteristics (including the first author, publication date, country, and mental health problems), (2) participant characteristics (including target sample, sample size, age, and gender), (3) intervention characteristics, (4) outcome measures, and (5) engagement and user experience.

Study results related to mental health outcomes were extracted in the form of means and SDs at baseline, postintervention, and follow-up when available, or, if not available, effect sizes. The study authors were contacted in case of missing or unclear information, and the study was excluded if they failed to provide the data.

It should be noted that in digital interventions, interaction frequency was usually used to represent dosage [66]. However, because different studies had different operational definitions of interaction frequency, to preliminarily explore the dose-response relationships of CAIs in mental health, we uniformly used the comparable data, average duration of interaction per day, to quantify the dosage in meta-regression.

The quality of the trials was independently evaluated by 2 authors using the Cochrane Collaboration tool [67] to assess the risk of bias. The Cochrane Collaboration tool evaluates 7 domains (sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective outcome reporting, and other risk of bias). For each domain, a rating of low (+), high (−), or unclear (?) was made for each trial. Only trials that received a low-risk rating on all 7 criteria were considered to have a low risk of bias. Trials were considered to have a high risk of bias if they were rated high in any bias domain other than performance bias, as blinding of participants and personnel is almost impossible in CAIs studies [68]. The risk of bias graph was drawn using Review Manager (version 5.4; The Cochrane Collaboration).

Given the differences at baseline, the change-from-baseline scores were computed for each group to represent the pre-post efficacy. For each comparison between the treatment group and control group, the standardized mean difference was calculated, as there were different scales from the included study to measure the same outcome.

We chose Hedges g as the effect sizes [69]. A positive g indicates that the CAIs condition had better outcomes than the comparison condition. If means and SDs were not reported, the effect sizes were converted via other available statistics (eg, d or η²). If data from both intention-to-treat and completer analyses were presented, the former were extracted and analyzed. We analyzed the pooled effect sizes of outcomes at postintervention as the short-term efficacy and the pooled effect sizes of outcomes at follow-up as the long-term efficacy.

Stata (version 15; Stata Corp) was used to perform the analyses. Heterogeneity between studies was quantified by calculating the I² statistic and Cochran Q statistic [70]. A random effects model was applied when substantial heterogeneity was observed (P<.05 or I²>50%); otherwise, a fixed effects model was used [71].

In a few studies, the same intervention condition was compared with >1 control condition (or vice versa), which may have artificially reduced the heterogeneity estimate and affected the pooled effect size, as these comparisons were not independent of each other. Thus, sensitivity analyses were run: only the comparison with the smallest effect size included in the meta-analysis or the largest effect size, which ensured that only 1 effect size per study was included in the analysis. We also performed an analysis only including low-risk articles to determine the pooled effect size after controlling for the risk of bias.

We conducted a subgroup analysis and meta-regression with the primary outcomes (depressive symptoms, generalized anxiety symptoms, specific anxiety symptoms, quality of life, general distress, and stress). Subgroup analyses were performed using Comprehensive Meta-Analysis (version 3; Bio-stat Inc) under a mixed effects model, which pools studies within a subgroup using a random effects model but tests for statistically significant differences between subgroups using fixed effects models [72]. As avatar does not fit the definition of the self-guided intervention, we pooled the effect sizes of the chatbot, ECA, and CA in VR to show the effect of self-guided CAIs in subgroup analysis.

Univariate random effects meta-regression used residual restricted maximum likelihood to measure between-study variance (τ²) with a Knapp-Hartung modification as recommended models [73].

We applied different methods to examine publication bias (funnel plot, Begg and Mazumdar rank correlation test [74], Egger regression test [75], and Duval and Tweedie trim-and-fill procedure [76]).

The 32 RCTs with 26 CAs were included, involving 6089 participants, with sample sizes ranging from 19 to 2668 (Table S2 in Multimedia Appendix 1). The average age was 36.32 (SD 13.44; range 21.40-71.47) years, and 75.41% (2922/3875) of the participants were women from the reported data. A total of 16 studies with CAs were classified as chatbot [28-43], 6 studies with CAs classified as ECA [44-49], 5 studies with CAs classified as CA in VR [50-54], and 5 studies with CAs classified as avatar [27,55-58].

Among the included studies, 9 were conducted in the United States; 6 were conducted in the United Kingdom; 3 were conducted in Sweden; 2 each conducted in Japan, Korea, Switzerland, and Canada; and 1 each conducted in Argentina, China, Ireland, the Netherlands, Germany, and New Zealand. A total of 8 studies recruited clinical samples, 14 recruited symptomatic samples, and 10 recruited general samples. In addition, 17 studies were based on cognitive behavioral therapy (CBT), and 15 studies were based on other theories such as patient-centered therapy, acceptance and commitment therapy, method of levels therapy, and problem-solving treatment. A total of 18 studies supported personalization and tailoring, and 14 studies did not support personalization and tailoring. Moreover, 18 studies supported emotional and empathic responses, and 14 studies did not support emotional and empathic responses. In total, 15 studies provided automatic reminders to engage, and 17 studies did not provide automatic reminders to engage. Nine studies directly aimed at depressive symptoms, 11 studies directly aimed at generalized anxiety symptoms, 6 studies directly aimed at specific anxiety symptoms, 2 studies directly aimed at quality of life, 5 studies directly aimed at general distress, 2 studies directly aimed at stress, and 4 studies directly aimed at other outcomes. Eleven studies had an intervention length of 0 to 4 weeks, 10 studies had an intervention length of 5 to 8 weeks, and 11 studies had an intervention length of >9 weeks. Seventeen studies had no follow-up, 7 studies had a follow-up length of 0 to 8 weeks, and 8 studies had a follow-up length of ≥9 weeks. Fourteen studies used passive controls, 8 studies used information or attentional controls, and 10 studies used active controls.

Attrition rates between baseline and postintervention measures were reported in 31 studies and varied widely from no attrition to 82.98% (2214/2668) of participants. More dropouts were from the control condition (2081/3583, 58.08%) than from the intervention condition (740/1888, 39.19%). The ration of the interaction reported in 16 studies ranged from 0.57 to 6.43 (mean 4.48, SD 2.11) minutes per day. Most studies (14/17, 82%) have reported good acceptability and usability of CAIs. Fifteen studies reported user experience, of which 8 performed thematic analysis on the feedback of participants.

Interrater reliability suggested substantial agreement between the raters for 7 domains of the Cochrane Collaboration tool (Cohen κ=0.86, 0.79, 0.67, 0.89, 0.93, 0.75, and 0.83, respectively). Eight studies were assessed as having a low risk of bias, 11 studies had some risk of bias, and 13 studies had a high risk of bias (Figures S1 and S2 in Multimedia Appendix 1).

The funnel plot of short-term effects (Figure S3 in Multimedia Appendix 1) and long-term effects (Figure S4 in Multimedia Appendix 1) and the results of Begg and Egger tests performed well on most mental health outcomes (Table S3 in Multimedia Appendix 1).

A total of 32 RCTs were found in our systematic review and meta-analysis to have evaluated the efficacy of CAIs in easing the symptoms of a range of mental health problems. With effect sizes ranging from g=0.24 to 0.62, most of which remained robust even after performing various sensitivity analyses, they were significantly better than control conditions in improving depressive symptoms, generalized anxiety symptoms, specific anxiety symptoms, quality of life or well-being, general distress, stress, mental disorder symptoms, psychosomatic disease symptoms, and negative affect. The long-term effects of CAIs on depressive symptoms, general distress, stress, mental disorder symptoms, and psychosomatic disease symptoms were statistically significant (g=0.16-0.39). More high-quality evidence is needed in the future to explore the long-term efficacy of CAIs.

Different mental health problems responded well to the 4 different types of CAs. Chatbot showed the largest effect size for generalized anxiety symptoms and quality of life, ECA showed the largest effect size for depressive symptoms and stress, CA in VR showed the largest effect size for specific anxiety symptoms, and avatar showed the largest effect size for general distress. This suggested that the efficacy of the different digital methods varied.

When compared with active controls, only the CAIs for depressive symptoms and general distress showed statistically significant effect sizes. Significant effect sizes were observed in the nonsymptomatic (depressive symptoms, generalized anxiety symptoms, quality of life, and general distress), symptomatic (depressive symptoms, generalized anxiety symptoms, and quality of life), and clinical samples (depressive symptoms, specific anxiety symptoms, and general distress).

Although there was no statistically significant correlation between the length of the intervention and its efficacy, we recommend extending it to better determine how clinical outcomes and nonclinical metrics interact during the intervention of CAIs.

CAs with advanced empathetic skills have improved user affinities and experiences [77,78]. In mental health CAs, the use of user profiles or user models to support personalized and adaptive features and the assessment of personalization is still in its infancy [79]. The results of this meta-analysis demonstrate how the use of personalization and empathic responses can significantly improve the efficacy of CAIs. In particular, empathic responses were linked to larger effect sizes for all mental health outcomes. This suggests that future technology and mechanism research on CAs may concentrate on these 2 capacities. A breakthrough in these 2 capacities will be necessary for CAs to function as competently as human therapists.

We included automatic reminders and theoretical orientation as moderators in the subgroup analyses, as in the earlier meta-analyses of smartphone interventions. In contrast to previous research [72,80], which concluded that studies offering engagement reminders were consistently associated with larger effect sizes, our study showed a stronger effect on depressive symptoms when the intervention did not include automatic reminders to engage. The same phenomenon was observed in the effect sizes of generalized anxiety symptoms, general distress, and stress, although it was not statistically significant. This may be because digital interventions such as internet-based CBT largely consisted of linear, structured psychotherapy modules [81,82], whereas more freedom for participants was offered in the context of CAIs [12], where interference and repetition would instead have an adverse effect on interest. We learned that the design of CAs needs to be succinct and empathic because, otherwise, the user’s interest will quickly wane, which is not conducive to the establishment of a stable working alliance between CAs and participants. Coupled with feedback for CAIs of participants in some studies, such as “process violations, repetitive content, misunderstanding, impersonality, not enough interactivity, and unnatural conversation,” we think that “maintaining engagement in the therapeutic process” will be the next area of focus for the development of CAIs for mental health problems rather than “attracting participation into the therapeutic process.”

As found in depressive symptoms and generalized anxiety symptoms, dosage is significantly positively correlated with the efficacy of CAIs so that a moderate increase in the frequency of interaction between CAs and participants is beneficial. Future CAs work to increase participants’ willingness to actively engage in conversation was supposed rather than only requiring them to accomplish a task in a passive manner.

There were certain limitations in this study that should be considered. First, a thorough search was conducted, prioritizing sensitivity over specificity because of the lack of standardized language in this field. Although some of the included studies treated a specific mental health issue as a secondary outcome or auxiliary outcome, the data were still extracted and used in the corresponding meta-analysis, and we used the broadest range of participants and comparators, which may have resulted in significant heterogeneity. Second, we chose the measurement instruments that most frequently corresponded to each of the mental health outcome to extract the data. We then used the standardized mean difference to remove the effect of dimensionality; however, it should be noted that different measurement instruments for the same mental health outcome are not exactly equivalent. As a result, if sufficient research has been conducted, it makes more sense to use a single measurement tool or to perform additional subgroup analysis for the measurement tool. Finally, in meta-regression, participant characteristics can only be included as covariates at the study or trial level for analysis, which may not accurately reflect the level of individual participants, leading to aggregate bias [83].

In conclusion, our study found evidence for the efficacy of existing CAIs for mental health problems and offered the most thorough summary of their clinical traits and nonclinical metrics. When compared with various control settings and across diverse groups, CAIs significantly improved a variety of mental health outcomes in the short term, but their long-term impacts were less than ideal. The performance and efficacy of 4 different forms of CAs (chatbot, ECA, CA in VR, and avatar) varied for various mental health problems. The efficacy of CAIs was strongly connected to 2 key facilitators: personalization and empathic response. Efficacy was not improved by receiving too many automated reminders to participate in CAIs. We also found a positive dose-response association.

In the postpandemic and digital eras, CAIs are likely to play a significant role and contribute significantly to the new health transformation. We still require more high-quality publications, particularly evidence of direct contrasts with guided digital interventions or web-based interventions. It is intended that multidisciplinary collaboration and integration continue to advance until the divide between theoretical mechanisms and technological development is effectively eliminated. Nevertheless, it is crucial to understand that CAIs are still in their infancy and have a long way to go before they can be widely used in clinical practice and reach their full potential within existing models of mental health care.