Digital mental health interventions (DMHIs) such as mobile apps, internet-based programs, and other technology-delivered initiatives have exploded onto the therapeutic landscape with estimates that 10-20,000 mental health apps currently exist [1-3]. Organizations have followed suit, and such interventions are widely implemented and promoted within workplace settings [4].

Emerging systematic review and meta-analytic evidence suggests that workplace-focused DMHIs that have been subject to trials produce small but significant reductions in symptoms of mental ill health among employees [4-7] and modest, potentially sustained increases in productivity and engagement in the workplace [8].

A critical finding from our 2022 meta-analyses is the significant heterogeneity observed between studies, with trial effect sizes ranging from –1.53 to +0.59 standardized mean differences [4]. This wide variability, including potential harm, underscores the need to understand the factors that influence intervention efficacy. Identifying these factors would enable developers to refine intervention designs and develop more targeted implementation strategies, with personalization of DMHIs seen as key [9].

While sample characteristics such as age, gender, and occupation, along with factors such as setting and context undoubtedly contribute to this outcome heterogeneity, and they are beyond the designer’s control. A systematic review [10] of smartphone apps for mental health found substantial variation in the content features, design, and implementation strategies. Characteristics such as the therapeutic approach, design of the intervention, and the content features are within the designer’s purview and could be optimized to enhance therapeutic effects.

This systematic review adhered to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) [18] guidelines (Multimedia Appendix 1) and was registered on PROSPERO (CRD42022323301). We aim to identify all published and unpublished, peer-reviewed, randomized controlled clinical trials of DMHIs targeted at employees that reported outcomes on a standardized mental health measure of depression, anxiety, or stress.

A systematic literature search was conducted in MEDLINE, PsycINFO, CENTRAL, and Embase from 2004 (when the first DMHI was identified) [7] until April 2024 using relevant keywords. Additionally, we manually searched selected journals and reference lists of included studies. Covidence systematic review software was used to manage the search, including automatic identification and removal of duplicates. The search strategy is provided in Multimedia Appendix 2. The search terms were developed from our previous systematic reviews [4,7].

The selection criteria for the study are as follows:

Two authors (ES and IC) independently screened titles, abstracts, and full texts. Discrepancies were resolved through consensus with the senior author (NG).

Risk of bias was assessed using the Cochrane Risk of Bias tool (version 2) [19]. Two authors (ES and IC) independently evaluated 5 domains: randomization, intervention deviations, missing data, outcome measurement, and result selection. Studies were categorized as “low risk,” “some concerns,” or “high risk” based on these assessments. Multiple intervention arms were evaluated individually.

Data were synthesized using a Bayesian multilevel meta-regression model. This approach accounted for the nested structure of the data (interventions within studies) and enabled estimation of pooled effect sizes, comparison of effects, and assessment of moderator impacts, all with accompanying credible intervals (CIs). Posterior distributions were used to interpret the strength of evidence, with evidence ratios (ER>3) indicating likely therapeutic benefit. Full model specifications and synthesis logic are provided in Multimedia Appendix 3.

All analyses were undertaken using R (version 4.2.0; R Foundation for Statistical Computing). Meta-regressions were performed using brms (version 2.17.0), an R interface to Stan (version 2.21.5).

We summarized the included studies, describing sample and intervention characteristics both overall and for each specific outcome. Pooled effect sizes (compared to placebo) for the efficacy of interventions for each specified outcome and their heterogeneity were estimated using a multilevel approach, as outlined by Harrer et al [20].

The meta-regression model was defined as:

In this model, represents the observed effect size of each study; θ denotes the true pooled effect size when all other moderators equal 0; β is a vector of model coefficients; X_k is a matrix of moderators and baseline differences; represents the within-study sampling error, which is modeled as a normally distributed random variable with known variance σ²ₖ derived from study-level data; and ζ_k represents true between-study heterogeneity with variance τ².

To explore the factors potentially influencing effectiveness, we conducted a moderator analysis. We calculated the proportion of between-study variance explained by moderators using the formula:

where the resid_intercept represents the residual between-study variance in the meta-analysis model, which included the intercept and baseline group difference only (Multimedia Appendix 3). The full model included the intercept and two sets of moderators: (1) baseline study sample characteristics and (2) intervention characteristics. Thus, R² represents the proportion of between-study heterogeneity explained by each set of moderators (sample vs therapeutic approach vs design vs content features).

We estimated the individual effect of each intervention characteristic on symptom reduction for each of the intervention characteristics (grouped by therapeutic approach, design approach, and intervention features). For each β, we determined the effect on symptom reduction, marginalizing over all other intervention characteristics. The average marginal effect of each characteristic (β) is presented to indicate the size and direction of the effect size. We also provided the 0.025 and 0.975 quantiles containing 95% of the posterior values as a 95% CI of the range of likely effects. Related to this is the pp of a “therapeutic effect,” that is, P(β<0) (pp), where a value over 0.95 arbitrarily indicates that specific moderator is highly likely to have a marginal therapeutic benefit (ie, distinct from a P<.05). Finally, the ratio of a therapeutic benefit over harm (benefit/harm; P(β_p<0)/P(β_p>0)) was quantified using the Bayesian ER as described by Kass and Raftery [21]. We interpreted ER>3 as indicating moderate evidence and ER>10 as strong evidence for a nonzero therapeutic effect, relative to the null. ERs are not interpreted as direct probabilities or odds, but rather as a relative support for the model including an effect compared to the model without an effect.

To evaluate robustness to prior specification, we conducted sensitivity analyses using alternative τ priors with wider and narrower distributions: inverse_gamma(0.25, 1), inverse_gamma(0.10, 1), and inverse_gamma(0.50, 2), in addition to the primary inverse_gamma(0.50, 1) prior. In the inverse gamma prior distributions, the shape (α) and scale (β) parameters are reported explicitly to avoid duplication of Greek symbols used elsewhere in the model. The shape parameter controls the peak density and tail weight of the distribution, while the scale parameter determines its overall spread. For each specification, we examined changes in the posterior mean and 95% CI of τ and in the posterior distributions of the regression coefficients (β).

The search identified 4822 titles (Figure 1), and no additional studies were identified through manual searches. After removing duplicates, a total of 4444 titles and abstracts were reviewed (ES and IC). In total, 4212 papers were excluded based on eligibility criteria, leaving 232 for full-text review. Seven authors did not provide the requested data, and 144 papers were excluded for various reasons (Figure 1). In total, 81 RCT studies met eligibility criteria and were retained for analysis [22-102]. Studies with multiple intervention arms were treated as individual interventions, resulting in 98 total interventions for analysis.

Of the included studies, 11 (12.7%) [31,33,35,42,58,84,93,95,101-103] had low bias risk, 4 (4.6%) [38,71,91,104] had some concerns, and 71 (82.7%) [16, 23-30, 32, 34, 36, 37, 39, 40, 43-57, 59-66, 68-70, 72-83, 85-90, 92, 94, 96-100, 105-109] had high bias risk. Main issues were deviations from intended interventions and missing outcome data, common in DMHI research. Notably, 50 (58%) were rated high risk due to using waitlist control groups, preventing user blinding (Multimedia Appendix 4) [16,23-40,42-66,68-109].

A visual summary of risk of bias assessments is presented in Multimedia Appendix 4, following the standard color convention of the Risk of Bias tool (version 2): green (low risk), yellow (some concerns), and red (high risk). The figure summarizes the 5 core bias domains for each included study.

We identified 81 RCT studies, evaluating 98 different interventions in 25,480 participants across various industries, including health care, insurance, management, information technology, telecommunications, education, government, marketing, retail, banking, and human resources. The studies were conducted in 18 countries: the United States (n=14, 17.28%), followed by Germany (n=13, 16.05%) and Japan (n=13, 16.05%). The United Kingdom contributed 10 (12.35%), while both Sweden and Australia accounted for 7 (8.64%) each. There were 5 (6.17%) studies from the Netherlands. A smaller number of studies came from Italy (n=2, 2.47%), Hong Kong (n=2, 2.47%), and broader Europe (n=2, 2.47%), as well as from Korea (n=1, 1.23%). Single studies were reported from Finland (n=1, 1.23%), China (n=1, 1.23%), Brazil (n=1, 1.23%), Singapore (n=1, 1.23%), and a combination of the United States and Canada (n=1, 1.23%).

Weighted mean ages were 41-42 (range 22.8-57.5) years, with female participation averaging from 55% to 60.6% (range 4.9%-100%). Multimedia Appendix 5 shows the distribution of intervention characteristics for each outcome, with “N” indicating the number of interventions per characteristic used for each outcome.

The most common therapeutic approach was CBT for stress, depression, and anxiety. Most interventions were designed by clinical experts or in consultation with clinical experts, and almost half included person support. There were proportionately fewer studies with lived experience co-design (stress: n=8, 11%; depression: n=10, 17%; and anxiety: n=5, 12%). The most commonly reported features were audio narration (stress: 51, 72%; depression: n=44, 75%; and anxiety: n=34, 83%), and a homework diary (stress: n=54, 76%; depression: n=44, 75%; and anxiety: n=30, 73%); however, few were tailored for symptoms (stress: n=2, 3%; depression: n=4, 7%; and anxiety: n=4, 10%).

This comprehensive study used a Bayesian approach to a systematic review of 81 RCTs encompassing 98 distinct DMHIs for stress, depression, and anxiety in the workplace. We evaluated the influence of study and intervention moderators on the heterogeneity and therapeutic effect observed in these studies. In this updated review, we confirmed previous findings that DMHIs yield small but positive effects in alleviating symptoms of mental ill health among employees in occupational settings [4-7].

Our analysis of between-study variance revealed intriguing patterns across different mental health outcomes. For stress interventions, we found that intervention characteristics explained a substantial proportion of the variance (82% combined), while sample characteristics account for only 21%. This suggests that for stress, the design and features of digital interventions play a more crucial role in determining the effectiveness than the study sample. Anxiety outcomes show a more balanced picture, with sample characteristics explaining 55% of the variance and intervention characteristics accounting for 67% when combined. This indicates that both who uses the intervention and how it is designed are important factors in anxiety treatment outcomes. Conversely, in studies where depression was the outcome, neither sample characteristics (27%) nor intervention characteristics (22% combined) explain a large proportion of the variance. This finding suggests that other unidentified factors may be playing a significant role in determining effectiveness of digital interventions for depression. Our previous review [4] showed that between-study heterogeneity in depression outcomes was lower, with only moderate levels of heterogeneity observed compared to high levels in both anxiety and stress. In this study, we observed a similar trend, where the observed variance between studies was lower in depression compared to stress and anxiety, suggesting that there may actually be less variation to account for in interventions focusing on depression.

The relatively low impact of sample characteristics on variance in stress and depression interventions is particularly noteworthy, challenging assumptions about the primacy of individual factors in treatment outcomes. Our findings validate the need for further exploration of intervention characteristics, especially for stress and anxiety interventions, where they explain a substantial portion of the variance.

The meta-regression analysis provided additional insights into the impact of various intervention characteristics on treatment efficacy. Notably, mindfulness-based approaches demonstrated the highest probability of therapeutic benefit for employees experiencing depression and anxiety, while stress management techniques showed the greatest promise for those dealing with stress. These findings highlight the importance of tailoring intervention approaches to these specific mental health concerns. Interestingly, while DMHIs have shown effectiveness in reducing stress symptoms, their efficacy in treating depression appears more nuanced [4]. Contrary to expectations, the incorporation of the well-established face-to-face clinical approach CBT, when delivered via DMHI platforms, showed a lower probability of benefit for depression compared to other methods.

The mixed results for depression treatment via DMHIs underscore the complex nature of this condition. Depression often requires more intensive personalized care that may be challenging to deliver solely through digital platforms [110]. For instance, digital mindfulness-based interventions for depression face challenges in providing personalized care [111] and replicating the human connection [112] found in face-to-face settings. These limitations may impact their effectiveness in addressing the complex, fluctuating, and nuanced needs of individuals with depression, who often benefit from real-time adjustments and supportive interpersonal dynamics. Drawing on decades of research and experience, hybrid solutions that combine face-to-face and digital treatments are likely to be the most effective, offering a balanced approach that can address diverse needs [113,114].

Explicit involvement of mental health experts in the intervention design was associated with a higher probability of benefit for stress reduction, aligning with research highlighting the value of expert involvement in enhancing the effectiveness and safety of mental health interventions [115]. However, such involvement did not lead to great benefit for depression and anxiety. While digital interventions offer promising avenues for mental health support, the depth and complexity of depression and anxiety may necessitate more comprehensive approaches outside of what experts may rely on in clinical care.

DMHIs co-designed by individuals with personal experience of mental health conditions did not demonstrate greater therapeutic benefit than those not co-designed for any outcome. The null effect of lived experience co-design may be attributed to the lack of current studies, with lived experience added to the design approach in a mere 10% of the studies but warrants more exploration, given (1) its emerging evidence in adjacent DMHI fields [116] and (2) further confirmation or rejection of these null findings. A final caveat to these findings may be that the ascertainment involvement of experts and people with lived experience depended on explicit mention in the papers and may have been misclassified.

As with most other meta-analyses and now meta-reviews [12,117], our results confirmed that the provision of adjunctive support had a high probability of increasing benefit across all 3 mental health outcomes: stress, depression, and anxiety. This finding underscores the importance of human support in the form of an e-coach for delivery of digital interventions, suggesting that a blended approach combining technology with personal guidance may be most effective [113,114].

A novel result from this study was the identification of potentially more efficacious features. Using as accepted ER>3 as indicative of useful features, we found that video use appeared beneficial transdiagnostically and that providing individuals with the mental health feedback scores leads to probable treatment benefits for the reduction of stress and anxiety. This aligns with the hypothesis that such feedback can increase awareness of one’s condition and motivate engagement in treatment [118]. By recognizing the severity of their symptoms, individuals may be more likely to adhere to treatment, potentially leading to improved outcomes [119]. Videos prove effective in delivering psychoeducation and support, making complex information more accessible through visual and auditory content, potentially improving understanding and retention [120]. Research by Berry et al [120] found video-based interventions particularly engaging and appealing to participants.

There were no other features that consistently stood out as beneficial, although there were strong effects for homework diaries and reminder texts for stress outcomes. Homework diaries and reminder texts features may help maintain user engagement and motivation throughout the intervention process, although we cannot explain why they should have an effect on stress but not depression and anxiety. It may be that such features have mixed effects in a study on novel digital mental health assessment tool, which revealed generally positive perceptions of reminders among users [121]. However, the same study also identified a potential drawback: some users found frequent reminders intrusive, suggesting the need for careful implementation. In the same manner, a review on supporting homework compliance in CBT apps suggested that homework can support in-the-moment self-assessment reflections. However, the review also noted that feedback on the inconvenience of doing homework may hamper a patient’s motivation [122]. These contrasting findings underscore the importance of thoughtful design in DMHIs. Future research should consider the optimal delivery of reminders, for instance, the frequency and timing of such reminders, as well as ways to make homework features engaging and convenient for users. Striking the right balance between effective intervention and user experience will be crucial for maximizing the benefits of digital mental health tools.

The study’s findings on mood trackers present an unexpected contrast to their widespread use in DMHIs. Despite being used in about 20% of the reviewed studies and often preferred by users [123], mood trackers showed a low probability of benefit and potentially detrimental impacts on anxiety and stress. This outcome is surprising, given their demonstrated effectiveness in previous RCT [124]. Several hypotheses could explain this result. Constant mood monitoring might lead individuals to focus excessively on negative emotions, potentially exacerbating symptoms. Additionally, the pressure to accurately track and report mood could create a sense of burden, contributing to increased stress and reduced intervention adherence. These findings highlight the complexity of designing effective DMHIs. Features that are intuitively appealing or unappealing may not always yield positive outcomes. Moving forward, developers and researchers may need to reassess the implementation of mood trackers in DMHIs, exploring ways to mitigate potential negative effects or considering alternative methods for monitoring emotional well-being.

This study offers valuable insights into improving workplace DMHIs for mental health, but it is crucial to acknowledge its limitations. Most included studies had a high risk of bias, potentially affecting the reliability of the findings. The high heterogeneity among studies might also have influenced results. Another key limitation is the use of the Bayesian approach, which only allows for estimating the probability of benefits rather than standardized effects. This means we can identify which features are likely beneficial but cannot quantify the exact magnitude of these benefits across different interventions and outcomes. The proportion of variance explained by the current set of moderators was substantially lower for depression (R²=0.29) compared to stress (R²=0.93) and anxiety (R²=0.83). This suggests that the factors influencing intervention effectiveness for depression may differ or be more complex and that the current moderators—largely focused on intervention type and features—may be insufficient for explaining variation in depression outcomes. Future work may benefit from exploring additional depression-specific moderators such as symptom severity, intervention dosage, or therapeutic intensity to better model these effects. This study is also limited by including only studies published in the English language.

The study found small but positive effects on employee mental health, with intervention characteristics explaining more variance than sample demographics, particularly for stress interventions. Mindfulness-based approaches were most effective for depression and anxiety, while stress management techniques excelled in stress reduction. The analysis emphasizes the importance of co-designing interventions with mental health experts and incorporating adjunctive support to enhance effectiveness. It also highlights the need for thoughtful implementation of features like reminders and homework as well as a critical reassessment of mood trackers, which showed potentially detrimental effects despite their popularity.

These findings emphasize the need for a more systematic approach to DMHI design, considering factors such as the specific mental health outcome, integration of professional expertise, balance of automated features and human support, and optimal implementation of engagement tools. Future research should focus on developing a comprehensive framework to guide the design, implementation, and evaluation of digital workplace mental health interventions.