This review complies with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [25]. We aimed to identify all published and unpublished, peer-reviewed, controlled clinical trials of eHealth interventions targeted at employees reporting outcomes on a standardized mental health measure of depression, anxiety, and stress. The systematic review protocol was registered on PROSPERO (International Prospective Register of Systematic Reviews; CRD42020185859).

A systematic literature search was conducted using MEDLINE, PsycINFO, Cochrane Register of Controlled Trials, and Embase electronic databases for relevant peer-reviewed articles in English of controlled trials and RCTs published from 2004 (when the first eHealth intervention was identified) [19] to July 2020. Keywords used related to “workplace,” “intervention,” “outcome,” and “study design.” An example of this search strategy is presented in Multimedia Appendix 1. The search terms were developed from our previous systematic review [19]. The tables of contents of the Journal of Medical Internet Research, Journal of Internet Interventions, Occupational and Environmental Medicine, and the Journal of Occupational and Environmental Medicine, as well as the reference lists of included studies, were manually searched.

We extracted the mean and SD of standardized measures of stress or distress, depression, and anxiety and the sample size (n) in each arm (intervention and control) at baseline and at each follow-up time point. When sufficient details were not reported, the authors were contacted. Additional data concerning the type of design, intervention and control details, participant characteristics, study duration, length of follow-up, organizational population, and general outcomes were recorded (see the Results section). Studies were also coded by intervention type: universal interventions targeting relatively healthy participants; indicated interventions targeting subclinical symptoms or syndromes; and tertiary treatments for explicit diagnoses.

Studies reporting results of the same intervention and sample in different papers (eg, a postintervention and follow-up study) were treated as 1 study, and we used the first follow-up outcome point in the analysis.

We categorized studies according to their mental health outcome measures: anxiety, depression, or stress. Studies with multiple outcomes appear in >1 mental health outcome result.

The risk of bias in RCTs was assessed using the revised Cochrane risk of bias tool for RCTs (RoB version 2.0) [27], in which 5 domains were independently evaluated by 2 authors: randomization process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of the reported result. Each domain was assessed for the risk of bias. Studies were graded as (1) “low risk of bias” when a low risk of bias was determined for all domains, (2) “some concerns” if at least one domain was assessed as raising some concerns but not to be at high risk of bias for any single domain, or (3) “high risk of bias” when a high risk of bias was determined for at least one domain or there were bias concerns in multiple domains [27]. If a study contained >1 intervention arm, each intervention was assessed as 1 study.

The summary ES was the SMD (calculated as Hedge g, with 95% CI) between the intervention and control groups for each outcome measure (anxiety, depression, and stress) [28]. A positive ES (SMD) indicated that the intervention was more efficacious than the control. The level of significance was set at P<.05 and 95% CIs. The magnitude of the effect was categorized as large (SMD>0.8), moderate (SMD 0.5-0.8), small (SMD 0.2 to <0.5), or trivial (SMD<0.2) [29]. Pooling of ESs across studies was performed using a random-effects model in Comprehensive Meta-Analysis (Biostat Inc; version 3). We detected and accommodated outliers in the meta-analysis using the random-effects variance shift model and the likelihood ratio test. As the different types of eHealth interventions and approaches used may result in large heterogeneity [30], between-study heterogeneity was quantified using τ² (variance of true effects) and further assessed using the Higgins I² statistic, which estimates the proportion of observed variance not because of sampling error. An I² value of 75% was considered large, 50% was considered moderate, and 25% was considered low [31].

We estimated the results separately for each mental health outcome measure (anxiety, depression, and stress) at the postintervention and first follow-up time points. Meta-regressions were performed for each mental health outcome to identify whether the intervention approach (eg, universal), type of intervention (eg, CBT), type of control (eg, wait-list), or intervention delivery method (web-based vs smartphone) moderated the observed effect.

To evaluate differences in effectiveness over time, we used only within-group ESs from the intervention arms for anxiety, depression, or stress outcomes. ESs were calculated by dividing the difference between pre- and posttreatment means by the pooled SD of the difference [32]. ANOVA was used to assess whether there were any differences between the mean ESs for each year. For a test for trends, the within‐group ESs with positive signs indicated improvement, and negative signs indicated worsening. This method allowed for the difference in effects over time to be accurately calculated, allowing us to explore whether interventions improved with technology.

To assess small study effects (in cases where at least 10 studies were available for analysis), we used a funnel plot for the overall effects and each subgroup analysis, which compared the outcome effects with their SEs. We used the Egger regression test to examine further asymmetry of the funnel plot [33] with statistical significance based on P<.10. In cases where at least 10 studies were available and a small study effect was found, we used a Duval and Tweedie trim and fill analysis to quantify the magnitude of the small study effect [34].

The search strategy identified 3411 titles (Figure 1). Of the 3411 articles, after the removal of 371 (10.88%) duplicates, 3040 (89.12%) titles and abstracts were reviewed by the authors (ES, IC, and MA). Of these 3040 articles, 2893 (95.16%) articles were excluded based on the eligibility criteria, leaving 147 (4.84%) articles that were potentially relevant to the research question. Their full texts were examined by 2 independent researchers (ES and MA), and discrepancies were decided by the senior author (NG). Of the 147 articles, 74 (50.3%) were excluded for reasons provided in Figure 1. Of the remaining 73 studies, data were missing for 8 (11%) studies. The authors were contacted, and 5 responded with relevant data [35-39]. A total of 3 authors did not respond or were unable to obtain data; thus, these studies were excluded from the meta-analysis [35,40,41]. Of the 73 studies, 4 (5%) articles reported on the same study at different follow-up time points and were merged [42-45], leaving 68 (93%) studies, of which 2 (3%) were excluded as they used face-to-face treatment for control groups [46,47]. The remaining 97% (66/68) of articles were identified as meeting the criteria for quality assessment. Of the 66 studies, 5 (8%) used 2 intervention arms [48-52], and 2 (3%) studies used 3 intervention arms [44,53]. In this case, the intervention arms were treated as individual trials reported as author names; year of publication; and the letters a, b, or c (eg, Smith [a]). The number of participants in the control group was split evenly as a comparative arm to ensure that participants were not counted twice. This process resulted in 75 trials (Figure 1) for the meta-analysis. Multimedia Appendix 2 [36-39,42-45,49-53] provides the list of references.

The individual studies included in the meta-analysis showed that, overall, 12% (9/75) had a low risk of bias, 3% (2/75) had ≥1 concern, and 85% (64/75) had a high risk of bias. Most of the bias was observed in deviations from the intended interventions and missing outcome data, which is not uncommon and is an area for improvement in the eHealth field. An important aspect common to behavioral interventions is that 55% (41/75) of the studies were rated as having a high risk of bias because of the use of a wait-list control group. As such, it was impossible for users to remain blinded (Multimedia Appendix 3 [36-39,42-45,49-53]).

All the included trials used web-based self-report measures to collect the baseline and follow-up data. Multimedia Appendix 4 provides an overview of the scales used and their prevalence.

This updated review aimed to synthesize the burgeoning literature on the efficacy of eHealth interventions for anxiety, depression, and stress outcomes in organizational settings and employee samples. We identified 52 new trial interventions published since the 23 identified in our prior review in 2017 [19], a tripling of the body of evidence. The systematic search identified 75 relevant trials for the meta-analysis, delivering eHealth interventions either on the web or via a smartphone, with a combined sample of 14,747 employees.

eHealth interventions reduced mental health symptoms immediately after use, with small positive effects observed in anxiety (Hedges g=0.26), depression (Hedges g=0.26), and stress (Hedges g=0.25), and data from trials with longer follow-up periods showed similar effects. These results are comparable with those of previous reviews, where small overall effects were found at the postintervention and follow-up time points [19,20]. These results imply that since 2017, the efficacy of eHealth interventions compared with control conditions reported in trials has remained unchanged, suggesting that the effectiveness of eHealth interventions does not seem to be improving over time. An analysis of the within-group effect observed in the intervention arms confirmed that since 2004, there has been no apparent systematic improvement in the effectiveness of these interventions. This is a surprising finding, given the enormous literature on methods to improve engagement with such interventions, greater penetration of technology to populations that do not access health care, and the increasing number of interventions delivered as apps, which reportedly improve access. All of these are commonly cited as factors favoring eHealth as a mode of intervention delivery or improving effectiveness.

The small but significant effect at follow-up suggests that eHealth interventions might have sustained positive effects on mental health. However, only half of the studies assessed follow-up outcomes, which may reflect a reporting bias. Supporting this, null, and in some cases, even negative, effects on mental health were observed in 27% (20/75) of the studies, and a greater proportion—60% (45/75)—did not report follow-up outcomes.

Less than one-fifth of all the trials used active controls. However, we found no significant differences in the efficacy of interventions tested against active versus passive controls overall or for any individual mental health outcome (Table 2). These findings do not confirm the digital placebo effect, which has been found in other eHealth reviews in clinical populations, where the ESs were lower in trials that used active controls than in those that used passive controls [24]. Given that most (47/75, 63%) of these trials were not conducted in clinically unwell populations but were delivered universally to employees, this contrary finding may have implications for the types of control recommended for different settings of eHealth trials and for future framework analysis or guidelines.

The primary moderator of efficacy appeared to be the content of the intervention, with stress management and mindfulness-based interventions being seemingly more efficacious than CBT-based interventions. This suggests that CBT, which was adapted from its intended use in a more personalized and clinical setting, may have less useful content, especially for use in organizations with universal delivery, than other approaches adapted for more universal or indicated tools such as mindfulness and stress management.

In our previous review, stronger effects were seen in eHealth interventions that were supported [19], and other studies have found increased effects when eHealth interventions were supported by trained mental health professionals [105]. The positive impacts of supported interventions may suggest that even in generally subclinical populations, a combination of eHealth and adjunctive support is the most effective in reducing mental health symptoms in employees.

Previous research has pointed to the importance of tailoring eHealth interventions to match individual user needs, as mental health symptoms differ from person to person [106]. However, this review did not support the notion that the tailoring of interventions provides any benefit to improving mental health. The limited evidence available suggests no greater efficacy when using tailored interventions than when using interventions delivered universally or for indicated or even unwell samples. In fact, 43% (3/7) of the tailored interventions showed potential for harm in at least one mental health outcome.

This study had some limitations. Significant heterogeneity was detected; however, this is not uncommon in this field or in meta-analytic research [24]. Differences across studies should be considered when interpreting these findings. It must also be acknowledged that some of the follow-up analyses were underpowered and that the findings should be interpreted tentatively. Furthermore, this review did not include any gray literature, and some authors did not provide data. Finally, the individual study pooled data calculated using the Comprehensive Meta-Analysis by Biostat Inc are not presented in this manuscript. Instead, we report the raw data of the pre-post means, SDs, and sample sizes in Table 1.

As we were unable to detect that eHealth interventions are improving over time, to design effective eHealth interventions, a better understanding of the factors that may influence efficacy is required. This study did not consider engagement and adherence and only considered a small number of potential moderating factors for each intervention. A subsequent framework analysis model is being undertaken to establish the potential beneficial or harmful features of the different types of interventions. This is an important future direction and requires further in-depth analysis as eHealth interventions have the potential to offer a range of novel self-management tools for employees with clinical and subclinical mental health conditions.

Another recommendation is to develop a standard framework for eHealth interventions to best understand the features that have therapeutic benefits and those that may potentially cause harm. This will guide developers to ensure that eHealth interventions are designed in the most effective manner. A framework may address the significant heterogeneity within studies; if the development standards are regulated against this framework, the differences in the features of the interventions and how they are delivered may be reduced. This research suggests that those evaluating interventions or designing protocols should carefully consider the level of support provided when interpreting reported ESs.

This review and meta-analysis confirms that eHealth interventions have a small but positive impact overall on reducing mental health symptoms in employees. There was significant heterogeneity between trials; however, overall, stress management and mindfulness interventions comprising in-person support appeared to be the most effective. Organizations should carefully consider the interventions delivered within the workplace; otherwise, they may not see long-term value in their return on investments. A substantial minority of intervention trials have demonstrated no efficacy, and a few may even be harmful. There is not enough evidence to make recommendations for the preferential use or development of therapeutically tailored interventions. Disappointingly, we found no evidence that despite the advancements in technology and the enormous resources in time, research, and finance devoted to this area for over a decade, better interventions are being produced. Hopefully, these small ESs do not represent the optimum outcome in organizational settings, and stakeholders and researchers should focus on improving effectiveness and efficacy or comparing and understanding the effects of current interventions. Despite the small effectiveness, this level of reduction could result in a large economic value in an organizational setting.