We developed a chatbot intended to screen and monitor anxiety and depressive symptoms and work-related burnout while providing evidence-based psychological interventions. User-centered development strategies have been adopted with evidence of improved user retention and satisfaction with the end product and increased chances of engagement in long-term real-world clinical settings [57]. Considering the preferences and recommendations of patients and health care workers presented in the protocol [56], the research team outlined the priority aspects of the digital tool as well as the psychological contents to be included. Its main functions included were as follows (Figure 1):

To consolidate the development and evaluate the feasibility and potential effectiveness of the intervention, the following phases were conducted sequentially.

Statistical analyses were conducted using IBM SPSS Statistics for Windows Version 28.0 (IBM Corp). The analyses of all subjects were considered until their last interaction with the chatbot. Sociodemographic and clinical data of the sample, as well as details on the general use of the intervention and the specific functions used, were evaluated using descriptive analyses. The Shapiro-Wilk test and visual inspection of Q-Q plots were used to assess whether continuous variables were normally distributed.

The main outcomes for potential effectiveness were changes in anxiety-depressive symptoms and work-related burnout measured with self-assessment scales using paired 2-tailed t tests or the Wilcoxon signed-rank test. Subgroup analyses were conducted to assess the influence of baseline symptoms and the effect of engagement on intervention effectiveness. In addition, to assess the relationship between use and effectiveness, we performed Pearson correlations on the number of times that modules were completed and the days of use with the change in clinical scales’ scores. The results of the MAUQ at the end of the first month of use (acceptability, usability, and satisfaction), as well as other objective-use UEIs (completion, adherence, compliance, and engagement) and feasibility, were presented using descriptive analyses and analyzed according to the definitions recently proposed by the International Society for Bipolar Disorders Big Data Task Force [72]. Statistical significance was set at 2-tailed P≤.05.

The PRESTO project was approved by the Clinical Research Ethics Committee at the Hospital Clínic of Barcelona (HCB/2020/0735). All data were collected and stored in encrypted and secure servers following the guidelines and standards of the 2018 European General Data Protection Regulation and were solely managed by project researchers. The project is in line with the ethical standards of European experts on personalized interventions and precision psychiatry [73]. All participants provided written informed consent before inclusion in the study.

A total of 40 HCs tested Vickybot simultaneously. They used different functions of the intervention, with all functions tested by at least 7 users. Data were transmitted and registered correctly, and the servers proved reliable with a rapidly increasing number of users.

A total of 17 HCs (n=13, 76% female; mean age 36.5, SD 9.7 years) simulated different clinical situations while using Vickybot. Table 1 shows the specific clinical simulations and respective functions of the interventions to be assessed.

All simulators signed up, performed the first self-assessment, and obtained a treatment plan with specific psychological modules. According to the respective simulations, 98.8% of the expected modules were recommended to the users (Table 1). Only 1 module was not offered in the respective treatment plan (user 4) because it had a lower-than-expected score on the PHQ-9. Furthermore, 59% (10/17) of users were offered “extra” modules, which were not expected in the simulation. This was mostly because of high scores on the clinical scales by simulators who were asked to report specific symptoms solely (users 9-16). The expected simulated severity of anxiety and depressive symptoms was achieved by 75% of the users (1-4) with 1 scoring below the expected severity (user 4). In total, 98.5% of the expected functions (Table 1) were tested by simulators according to each clinical situation with the expected outcomes. All simulators showed a progressive reduction in scales’ scores according to the clinical simulation of symptom reduction. This was not the case for 1 user (user 17) who was instructed to simulate “erratic behavior.” Suicidal alerts were correctly activated by expected users (4 and 7) and received by the research team. Overall, 53% (9/17) of users set up “weekly objectives,” of which 67% (6/9) rated their daily completion. Moreover, 65% (11/17) of users set reminders, and 53% (9/17) of users registered an audio file after self-assessment.

A total of 65% (11/17) of HCs responded to the MAUQ after the simulation phase, with a mean score of 6.39 (SD 0.36) on a scale from 1 (strongly disagree) to 7 (strongly agree). All simulators (34/34, 100%) agreed with 14 of 18 statements of the MAUQ questionnaire. Most patients (72%-90%) agreed with the remaining 4 of the 18 statements. Vickybot showed high usability, satisfaction, and acceptability among the simulators. The most recurrently perceived weak points to be improved were the lack of reminders, the user’s lack of capacity to configure or personalize them, the lack of incentives to answer the self-assessments or access the psychological modules, and not being understood by the chatbot many times.

This study details the development and evaluation of the feasibility and potential effectiveness of a chatbot for anxiety, depressive symptoms, and work-related burnout among both patients from PC and health care workers. This intervention forms part of a larger project that aims to generate a digital platform (PRESTO) combining machine-learning severity prediction models with a smartphone-based app for screening, monitoring, and delivering psychological interventions [56]. The smartphone-based intervention has been envisaged to cover some of the most concerning demands from the PC system and also to provide personalized help to health care workers, as both have been affected by the COVID-19 pandemic. These multiple purposes require stepwise and solid growth as well as the collaboration and coordination of specialists in different areas.

The chatbot was useful in screening anxiety and depressive symptoms and work-related burnout in patients from PC and health care workers using self-administered scales. Although there were no statistically significant reductions in anxiety or depressive symptoms in the follow-up self-assessments, there was a significant reduction in work-related burnout (present in >60% of our sample). Similar to other digital solutions [74,75], Vickybot pointed toward potential effectiveness in reducing work-related burnout. Further research with increased statistical power is required to confirm these findings. There is a need to tackle this growing issue with measures from occupational and health care systems, and digital tools could offer a potential solution to this highly prevalent problem. Finally, the chatbot allowed for accurate detection and prompt intervention in emergency situations (suicidal thoughts).

It is known that health care workers and people referred to specialized mental health care from PC are distinct in several aspects, including medical literacy and work-related conditions. Despite the differences between both groups, they also share several common aspects, such as a high prevalence of depressive and anxious symptoms. Even though the causes of work-related burnout are particularly and highly specific (ie, stress and lack of support at the workplace), its manifestations include mostly anxiety and depressive symptoms such as fatigue, insomnia, and sadness. Moreover, even though health care workers share a common work environment, they are highly heterogeneous in their backgrounds and work situations, including people from medical staff, nursing staff, nursing assistants, administrative workers, management, cleaning staff, security, and logistical support. In fact, during the development phase, these considerations were debated and taken into account, but we found that despite the distinct characteristics between the 2 groups, their problems and preferences for a mental health app were highly similar. In this study, subanalyses between health care workers and PC patients regarding UEIs and potential effectiveness outcomes were performed, but no differences were found. This finding reinforces the hypothesis that these 2 groups have more common mental health problems than specific differences.

It should be noted that severe anxiety and depressive symptoms (GAD-7 or PHQ-9 ≥15) were observed in 44% (15/34) and 42% (14/34) of the patients, respectively, and 24% (8/34) presented both severe anxiety and depression simultaneously. In addition to the reduced sample size and low engagement, the high severity of symptoms in >40% of the sample may explain the lack of reduction in anxiety and depression symptoms. Digital interventions have only been found to be effective in the reduction of mild-to-moderate depressive symptoms [41], but not for severe symptoms. Therefore, consideration of the characteristics of the target population is of utmost importance when designing digital interventions. In our case, we managed to identify patients with severe anxiety and depression, which, it could be argued, are probably not suited for this kind of intervention, as they may require face-to-face visits with mental health specialists. In this sense, according to the literature, chatbots may be useful screening tools for mental health symptoms by providing a “humanized” environment while preserving anonymization [76,77].

Detecting patients with high severity is a core problem faced by overloaded PC systems. Nowadays, the prioritization system has been shown to be ineffective [78], and digital solutions that can screen for severity may be suitable for solving this problem. Vickybot was useful and accurate in detecting both patients with higher severity in addition to people with suicidal thoughts, while also offering on-time tools for immediate help (ie, emergency services) and alerting the research team to evaluate and propose tailored interventions. Suicide is probably the most striking unsolved mental health problem [79], and COVID-19 consequences are likely to worsen this situation [80-83]. Suicidal behaviors are the most alarming emergency situations that can be encountered; thus, tools that allow prompt detection and immediate interventions are crucial. Digital solutions may fulfill these requirements [84], with some interventions already showing positive evidence [85]. In fact, because this is a very sensitive issue, consensus statements on how to tackle suicide with digital technologies have recently come to light [86], and conclusive remarks regarding the future consensus on the applicability of digital solutions in mental health in both research and clinical practice are yet to come.

Some smartphone-based interventions have shown efficacy in outcomes such as well-being, quality of life, or perceived stress, which are considered more important than symptom reduction by patients [87]. Likewise, interventions such as psychoeducation empowered patients with strategies to cope with symptoms. As such, even when symptoms are not reduced, enhancing patients’ perceptions of control and self-management are key interventions for improving their quality of life [88-90]. Studies on digital health interventions should therefore focus on these parameters and not only on symptom reduction to gain further insight into their effectiveness and real-world application.

Moreover, smartphone-based solutions usually include functions such as self-assessment of symptoms, longitudinal monitoring, detection of emergency situations (eg, suicidal thoughts), and specific tools based on psychological therapies (eg, CBT or mindfulness). It is still not clear which of the former functions are more suitable for each patient profile in terms of their severity of anxiety and depression, specific symptom presentation, or sociodemographic characteristics [91]. The subanalyses and correlations performed in this study showed a nonsignificant trend toward a reduction in anxiety and depressive symptoms and work-related burnout with greater use of the chatbot. However, the small sample size in this longitudinal assessment may have limited the results. Future studies with increased statistical power are required to shed light on the remaining unanswered questions.

Regarding UEIs, subjective perceptions of use (acceptability, usability, and satisfaction) were highly positive, both for HCs during the simulation phase and for patients during the feasibility and effectiveness studies. Patients especially valued the ability to perform mental health self-assessments and personalization of treatments, as well as the chatbot’s ability to detect urgent situations and offer immediate help. This contrasts with the low objective-use metrics (completion, adherence, compliance, and engagement). This was probably because the chatbot had a rigid and sequential structure that users had to follow (based on the usual clinical evaluation and treatment procedures), thus lacking flexibility of use, which is key to user retention. This was shown by the initial use of the chatbot on day 1, with all users completing the first self-assessments and accessing at least some modules of the treatment plan, followed by a gradual reduction using the chatbot, leaving many modules incomplete. Furthermore, some modules were not accessed by any user. Other factors that may have lowered engagement outcomes are probably the lack of incentives offered by the intervention (eg, paucity of personalized reminders) or the limitations of the natural-language understanding of the chatbot, as stated in the comments of the simulation phase. A decrease in engagement in the use of digital tools over time has been reported in the literature [48,92,93]. For instance, studies with similar digital interventions reported equivalent results on engagement, with most participants only using the app once or twice [94], or only 20% of participants providing the required information in the app during the study period, with a gradual loss of entries after day 1 [95]. This suggests that engagement is a widespread problem with digital interventions and that it is key to develop strategies to mitigate it. Interventions designed for brief interactions have shown higher engagement outcomes [96]. This suggests the need to adapt smartphone-based interventions to the ways people use these technologies (usually briefly and concisely) [41]. On the basis of users’ comments and the evidence in the literature, we propose that future versions of the chatbot should aim to be more flexible by offering users alternative flows of use at all times and also include gamification with user incentives, adding personalization strategies to the psychological modules and reminders, and enhancing the chatbot’s interpretation and response capacities with NLP techniques.

First, the limited sample size of the feasibility and effectiveness study, the low engagement with the intervention, and the high severity of anxiety and depressive symptoms of many of the patients included may have precluded the possibility of finding the potential effectiveness of the intervention in reducing anxiety and depressive symptoms. Accordingly, further research with increased sample sizes, particularly of patients with mild-to-moderate anxiety and depressive symptoms, and longer follow-ups are required to determine the effectiveness of the intervention.

Second, we did not assess PTSD symptoms, which are prevalent among health care workers [23], and other relevant outcomes, such as well-being, quality of life, and perceived stress [87]. We suggest that future studies consider these aspects to achieve a holistic perception of the intervention by users.

Third, although Vickybot succeeded in cross-sectionally screening and establishing the severity of anxiety and depressive symptoms and work-related burnout, it failed to provide follow-up (biweekly) information on the evolution of most of the sample. Notably, obtaining cross-sectional information on symptom severity is key for successful triage in PC settings and for identifying emergency situations (eg, suicidal thoughts) in which the intervention was useful. However, the key goal of the PRESTO project is to provide self-management tools for people with anxiety and depressive symptoms. Moreover, only 26.2% (34/130) of patients who received the SMS invitation signed up and performed the first self-assessments. This represents a loss of almost 3 of 4 potential participants, which is in line with other studies showing similar losses of potential participants in the initial phases [94]. We believe that this was because of access and download difficulties, hence, the future versions of the chatbot should simplify this process as much as possible. To achieve this, in-app sign-up (eg, ensuring all steps can be conducted within the app itself) and reducing the amount of data required from participants before access may provide a solution to reduce the complexity of this process. Altogether, accessibility to the chatbot and its functions need to be redesigned considering user preferences while also keeping in mind the relevance of monitoring patient evolution.

Finally, there is still a lack of consensus on how to assess the feasibility of smartphone-based interventions, especially chatbots [88,97]. In addition, UEIs still require consistent and replicable standard definitions and valid measures with solid thresholds [98]. In this study, we applied the definitions and metrics recommended by the International Society for Bipolar Disorders [72]. Although these guidelines are based on digital interventions for bipolar disorder, most concepts may be applicable to other types of interventions. In recent years, platforms and databases of mental health smartphone-based interventions have been developed [99,100], including information on the characteristics of the intervention, functions included, population for whom they are directed, evidence-based validation, professional support, and expert reviews. Future directions point toward international expert consensus, providing homogeneous valid indications on digital solutions’ regulation [101] and clinical validation [102].

Future lines of the PRESTO project include (1) enhancing the chatbot NLP component, including natural-language understanding and generation capacities; (2) integrating the chatbot into a smartphone app to facilitate accessibility and incorporate the aforementioned functions; (3) performing focus groups with patients and professionals to detect needs and points of improvement to increase engagement; and finally (4) integrating the app into a digital platform to be implemented in the PC system.

The chatbot was useful in screening for the presence and severity of anxiety and depressive symptoms. Although anxiety and depressive symptoms were not significantly reduced, there were significant reductions in work-related burnout on follow-up self-assessments, thus suggesting the potential effectiveness of Vickybot. Emergency situations were accurately identified and prompt interventions with successful outcomes were provided. Subjective perceptions of use (acceptability, usability, and satisfaction) were high, in contrast to low objective-use metrics (completion, adherence, compliance, and engagement), and the feasibility of the intervention was moderate. Our results are promising but suggest the need to adapt and enhance the smartphone-based solution to improve engagement. Finally, a consensus on how to report user engagement indicators and validate digital solutions, especially for chatbots, is required.