Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 31.08.20 in Vol 22, No 8 (2020): August

This paper is in the following e-collection/theme issue:

    Review

    Effects of Internet-Based Cognitive Behavioral Therapy in Routine Care for Adults in Treatment for Depression and Anxiety: Systematic Review and Meta-Analysis

    1Department of Clinical Psychology and Psychotherapy, Friedrich-Alexander University Erlangen-Nuremberg, Erlangen, Germany

    2GET.ON Institute GmbH, Hamburg, Germany

    3Department of Clinical, Neuro-, & Developmental Psychology, Faculty of Behavioural and Movement Sciences, VU Amsterdam, Amsterdam, Netherlands

    4Amsterdam Public Health Research Institute, Mental Health, Amsterdam, Netherlands

    5Department of Global Health and Social Medicine, Harvard Medical School, USA, Boston, MA, United States

    6Department of Clinical Psychology and Psychotherapy, Institute of Psychology and Education, Ulm University, Ulm, Germany

    7eCentre Clinic Department of Psychology, Macquarie University, Sydney, Australia

    8Community Mental Health Centre GGZ inGeest, Amsterdam, Netherlands

    Corresponding Author:

    Anne Etzelmueller, Dipl-Psych

    Department of Clinical Psychology and Psychotherapy

    Friedrich-Alexander University Erlangen-Nuremberg

    Nägelsbachstr 25a

    Erlangen, 91052

    Germany

    Phone: 49 9131 85 67575

    Email: anne.etzelmueller@fau.de


    ABSTRACT

    Background: Although there is evidence for the efficacy of internet-based cognitive behavioral therapy (iCBT), the generalizability of results to routine care is limited.

    Objective: This study systematically reviews effectiveness studies of guided iCBT interventions for the treatment of depression or anxiety.

    Methods: The acceptability (uptake, participants’ characteristics, adherence, and satisfaction), effectiveness, and negative effects (deterioration) of nonrandomized pre-post designs conducted under routine care conditions were synthesized using systematic review and meta-analytic approaches.

    Results: A total of 19 studies including 30 groups were included in the analysis. Despite high heterogeneity, individual effect sizes of investigated studies indicate clinically relevant changes, with effect sizes ranging from Hedges’ g=0.42-1.88, with a pooled effect of 1.78 for depression and 0.94 for anxiety studies. Uptake, participants’ characteristics, adherence, and satisfaction indicate a moderate to high acceptability of the interventions. The average deterioration across studies was 2.9%.

    Conclusions: This study provides evidence supporting the acceptability and effectiveness of guided iCBT for the treatment of depression and anxiety in routine care. Given the high heterogeneity between interventions and contexts, health care providers should select interventions that have been proven in randomized controlled clinical trials. The successful application of iCBT may be an effective way of increasing health care in multiple contexts.

    J Med Internet Res 2020;22(8):e18100

    doi:10.2196/18100

    KEYWORDS



    Introduction

    Depressive and anxiety disorders are common mental health problems associated with significant suffering, impairment, and reduction in the quality of life [1,2]. Both disorders lead to considerable socioeconomic costs through decreased work productivity and higher utilization of health care services [3,4].

    Despite the proven effectiveness of psychotherapy in the treatment of depression and anxiety [5], the provision of evidence-based treatments depicts a constant challenge given the barriers such as the shortage of treatment, uneven distribution of trained providers, delayed treatment provision, and inadequacy of treatment [6,7]. Furthermore, research on patients’ preferences has shown that many do neither make use of psychotherapeutic treatments nor do they receive psychopharmacological treatment [7]. Using the internet to provide psychotherapeutic interventions may increase the coverage of usual care services [8,9] by providing highly accessible and scalable interventions reaching people who cannot be reached otherwise. Recent research suggests that internet-based cognitive behavioral therapy (iCBT) with therapeutic guidance is effective for the prevention [10,11] and treatment [12-15] of common mental disorders. Systematic reviews on studies were also able to show comparable effects to face-to-face treatments in adults [16,17]. In a recent meta-analysis, Romijn et al [13] showed that iCBT interventions for anxiety disorders can also have significant effects obtained in trials implemented in clinical care. They also found that effects were smaller in samples recruited in clinical practice than in samples recruited with an open recruitment method compared with waitlist-control groups [13], which raises the question of the effects of iCBT when implemented in routine practice.

    Although randomized controlled trials (RCTs) are considered the gold standard in exploring the efficacy of mental health interventions, the idealized and controlled nature of these trials limits the generalizability of findings to routine care populations [18]. RCTs maximize the internal validity, to ensure that the effect found can be attributed to the investigated intervention [19,20]. Thus, RCT findings are restricted by controlled protocols, explicit eligibility criteria, and patient recruitment and randomization procedures. RCTs provide a highly structured environment, which is considered to possibly have an adherence-fostering effect [21,22]. The efficacy derived from RCTs of internet-based interventions might be overestimated for what can be expected when implementing in routine care, limiting the knowledge base for routine clinical practice [20].

    Hence, after establishing the efficacy of an intervention and its subsequent implementation, the so-called phase IV clinical trials should follow investigating benefits when implemented as well as potential negative effects implemented [23,24]. Thus, the investigation of the effectiveness of iCBT under routine care conditions is an important part of the evaluation of these services before wide-scale adoption.

    Andersson and Hedman [25] reported on the effectiveness of iCBT within four controlled trials and eight open studies for a multitude of mental health problems, indicating that it might be possible to replicate the findings of controlled efficacy trials on guided iCBT in clinical practice. However, in that review, both routine care and RCTs were included, and only eight studies reported effects when the service was delivered under routine conditions. Recently, Andrews et al [15] reported the results of computer-based treatments of depression, panic disorder, generalized anxiety disorder, and social phobia in randomized trials. They also identified eight studies on internet-based treatments in routine clinical practice when delivered outside of a randomized trial reporting an average effect size of g=1.07 across all 4 disorders [15]. However, since then, many more studies have been published. In addition, this review did not specifically try to identify nonrandomized trials, possibly leading to unidentified articles. Additionally, they did not provide disorder-specific results, specific results on guided treatments by mixing guided and unguided treatments, and did not investigate the acceptability and potential negative effects.

    The aim of this study was to examine the effects of guided iCBT for the treatment of depression and anxiety under routine care conditions on symptom change, acceptability (uptake, participants’ characteristics, adherence, and satisfaction), and predictors of negative effects (deterioration and side effects).


    Methods

    We report this meta-analysis in accordance with the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines (Multimedia Appendix 1) [26]. This meta-analysis was registered at international prospective register of systematic reviews (PROSPERO; trial registration: CRD42018095704).

    We searched PubMed, PsychINFO, and the Cochrane library. We used index terms and text words associated with depression and anxiety, internet interventions, and routine care (for a full search string, the reader is referred to Multimedia Appendix 2). Furthermore, we contacted experts in the field to ask whether they were aware of the studies that we did not identify through our systematic literature searches. Furthermore, we conducted reference tracking on the identified studies and previous meta-analyses in the field [5,14,15,27]. The resulting hits of our literature searches were screened on titles and abstracts by 2 independent reviewers (AE and CV). Studies considered as potentially relevant were screened on full text by the same reviewers independently. In case of disagreement, the opinion of a third senior reviewer (DE) was sought.

    Inclusion Criteria

    We included studies that (1) examined the effectiveness of a guided or blended iCBT in (2) treating adults with depressive and/or anxiety symptoms (3) under routine care conditions (4) in a pre-post design. We followed the inclusion of adults and older adolescents (aged >16 years) within the treatment provision for adults, as reported in the original studies.

    We defined routine care studies as effectiveness studies, which were conducted as nonrandomized clinical trials in settings equal to or representative of routine practice [28]. The definition of routine care differs between countries and health care systems and describes the established way of working at the time of the original study. Depression and anxiety symptoms had to be established based on cutoff scores on self-report outcome measures, clinical diagnosis, or expert opinion. The definition of anxiety symptoms is based on the Diagnostic and Statistical Manual of Mental Disorders IV classification criteria for anxiety disorders. Furthermore, the interventions were considered as guided when the guidance was related to the therapeutic content [29] and as blended when the internet-based intervention was combined with face-to-face elements in one integrated standardized treatment protocol [27]. Guidance can be delivered via email, a secure message system, telephone, or face-to-face contact and via video or face-to-face contact in blended treatments. Finally, both disorder-specific and transdiagnostic interventions (targeted at both depression and anxiety simultaneously) were included.

    Exclusion Criteria

    We excluded studies that did not (1) focus primarily on anxiety or depression or (2) provide sufficient data for the calculation of the effect sizes. Studies were also excluded if (1) the service had only been provided as part of a research study, (2) the study could be considered as a feasibility or pilot trial, or (3) patients were randomized at an individual level. However, cluster randomized trials were considered eligible, in which randomization took place not on an individual level but, for example, on a health care institution level. For the definition of feasibility and pilot trials, we followed the NIHR Evaluation, Trials and Studies Coordinating Centre definition of pilot and feasibility trials [30], as recommended by Arain et al [31]. Feasibility trials were defined as “pieces of research done before a main study” (designed around the research question “Can this be done?”), and pilot studies are defined as a version of the main study that is “run in miniature to test whether the components of the main study can all work together” [31]. Additionally, we only included studies published in English, German, or Dutch language.

    Data Extraction

    We extracted data related to study and iCBT service–related characteristics, acceptability, effects on symptom change, negative effects, and data related to the risk of bias of reported results.

    Study characteristics included the year of publication, the country in which the study was conducted, the year of data collection, sample size, eligibility criteria (establishment of depression and/or anxiety diagnosis at baseline [standardized clinical interview, cutoff on standardized questionnaire, and clinical judgment], inclusion of severe cases [yes/no], and exclusion of cases with suicidal ideation [yes/no], and approach to data analysis [ITT/completer]).

    iCBT service–related characteristics included intervention name, the symptoms targeted (depression and/or anxiety), if it was a blended treatment (yes/no), evidence base for the used intervention (positive results based on at least one randomized clinical trial [yes/no]), and whether it was a symptom-specific or transdiagnostic treatment. We also included the recruitment pathway (open community, clinical referral, and both), the number of planned intervention modules, guidance focus (content-focused, motivational-focused, and administrative-focused), guidance delivery format (synchronous vs asynchronous, within the treatment platform vs outside, eg, by email), and guidance moment (as a reaction to an action of the participant [eg, after the participant finished a session, as a reaction to a nonresponse] or planned in different intervals [eg, weekly or biweekly]). Furthermore, we included guides’ professional training (psychotherapist, psychiatrist, general practitioner [GP], psychologist, psychological registrar, nurse, coach [with lived experience]), training of professionals in iCBT (yes/no), supervision of professionals by a trained clinician (yes/no), the planned and actual intensity of guidance in minutes, and if there was a guidance manual provided (yes/no). Additional information on whether a standardized procedure in the case of symptom deterioration and crisis (yes/no) has been established was included.

    Acceptability data were extracted with regard to uptake (the number of people screened for the service, people included, and participants starting the treatment), patient characteristics (age and gender), average symptom severity at baseline, adherence (ie, number of completed modules), mean treatment duration in weeks, and participant satisfaction. Negative effects were extracted with regard to average effects on symptom deterioration, other side effects, and reports of specific subgroups at risk for symptom deterioration.

    Two reviewers (AE and CV) extracted the data independently, and data sets were merged. Differences and points of uncertainty were discussed and checked by returning to the original article and in some cases to the authors of the respective article.

    Risk of Bias Assessment

    Assessing the quality of naturalistic observational studies is challenging as there is no widely accepted tool in doing so [32]. Moreover, established guidelines for the quality assessment of nonrandomized trials are only partially applicable, as they assume comparisons of interventions (Risk Of Bias In Nonrandomized Studies of Interventions-I [33]). Thus, in this study, we selected and adapted criteria from two quality assessment tools [33,34] and adapted them to this study’s purposes to evaluate the risk of bias of the included studies. For the present risk of bias assessment, we discussed the aforementioned assessment tools among all coauthors of this manuscript and derived the analysis criteria described in Multimedia Appendix 3 [35]. As a result, we evaluated (1) researcher allegiance (defined as the first or last author of the study also being the first or last author of the intervention development or efficacy paper), (2) confounding introduced by patients’ participation in other treatments, (3) confounding introduced by significant confounding variables identified within the individual study (meaning any predictors included such as age, guidance, or recruitment pathway), (4) selection bias introduced by the study population (ie, have the studies only reported on completer data), and (5) selective outcome reporting in comparison with the study protocol or diagnostic measures administered as mentioned in the original studies’ methods section. A description of the risk of bias assessment and its operationalization can be found in Multimedia Appendix 3. With regard to Researcher Allegiance, we chose the above definition after consideration among the authors and evaluated a study as at high risk of researcher allegiance when the first or last author of the study was also involved in the development of the treatment manual of the psychotherapy involved or the reporting on the interventions’ efficacy. Although the validity of other indicators has been questioned, the involvement of a researcher in developing the treatment under investigation can be considered a valid indicator of potential researcher allegiance [36].

    Two reviewers evaluated the quality of the included studies independently (AE and CV). Any disagreement between reviewers was solved by a thorough discussion. If the disagreement could not be resolved, a third senior reviewer was consulted (DE).

    Statistical Analysis

    Our primary outcome was the reduction of depressive or anxiety symptoms from pre- to posttest assessment. We calculated the difference in depression and anxiety symptoms between pre-post assessment divided by the weighted, pooled standard deviation (Hedges’ g). We have chosen Hedges’ g because it allows for small sample size bias correction [37]. As we expected considerable heterogeneity among the studies, we used the random effects model. As a rule of thumb, effect sizes of 0.8 can be viewed as large, 0.5 as moderate, and 0.2 as small [38]. In our main analysis, we included mixed depression and/or anxiety studies into the separate depression and anxiety data sets. Statistical analysis was conducted using the Comprehensive Meta-analysis program (version 2.2.2), and pooled proportions were calculated with R [39] package meta [40].

    To calculate heterogeneity, we used the I2-statistic and its 95% CIs as an indicator of heterogeneity in percentages. Heterogeneity was interpreted as low, moderate, and high when 25%, 50%, and 75%, respectively.

    We also included the correlations of the used pre-post measures using the mean of 0.76, where none was provided for depression and 0.59 for anxiety following the study by Balk et al [41]. We also conducted sensitivity analyses for correlations set to 0.00, 0.75, and 0.99 to examine the robustness of our findings [42]. We also calculated the prediction interval, which estimates where the true effects are to be expected for 95% of similar studies that might be conducted in the future [43].

    As we expected high heterogeneity, we conducted several subgroup analyses to investigate its possible sources. The examined subgroups were related to the method of analysis, time to post assessment, recruitment pathways, disorders, guidance moment (specific timing or as a reaction), guidance modality (email, message, and synchronous), guide profession (with or without specific CBT training), supervision provided (yes/no), guide training provided (yes/no), intervention manual provided (yes/no), approach to data analysis (ITT/completer), and diagnostic method (interview/questionnaire). Subgroup analyses were only carried out with regard to the effects on symptom change. We used the mixed effects model, testing pooled studies within subgroups with random effects models while testing for significant differences between those subgroups with fixed effects models. We only conducted subgroup analysis if the number of studies per category was not less than three. If necessary, we combined predefined subgroups to achieve the necessary group size.

    Finally, we conducted meta-regression analyses for the continuous variables, examining the duration of the treatment as a predictor of treatment outcome as well as guidance time, number of contacts, number of sessions completed, and the percentage of treatment completers.

    Regarding uptake, we calculated the proportion of (1) included people based on the number of people screened, (2) starters based on the number of people being screened, and (3) starters based on the number of people included. Adherence was analyzed by calculating the percentage of modules completed based on the average number of sessions that were completed by the participants divided by the planned total number of sessions. We also coded the percentage of intervention completers for a 100% completion rate. Additionally, we pooled the age and gender distribution as well as participant satisfaction extracted from original studies. Furthermore, we pooled the percentage of individuals reported to show symptom deterioration (defined as a negative reliable change in the reported outcome), the deterioration rates, reported in the original study.

    Publication bias was examined by inspecting the funnel plot [44] and conducting the Egger test of the intercept with a one-tailed significance level of α=.05 [45]. In addition, we used Duval and Tweedie’s trim and fill procedure [46] to adjust the effect size for missing studies.


    Results

    Study Selection

    A systematic literature search was performed on January 30, 2019. This search resulted in 25,447 citations. After removal of duplicates, 19,316 citations remained for the title and abstract assessment and 174 after the exclusion due to title and abstract. A total of 17 studies fulfilled the eligibility criteria. The full references of the included studies are listed in Multimedia Appendix 4 (Etzelmueller et al, unpublished data, 2020) [47-63]. The study selection process is illustrated in Figure 1.

    Figure 1. Study inclusion.
    View this figure

    Study Characteristics

    Table 1 presents the characteristics of the included studies. Seventeen studies (n=12,096 participants) reporting on the outcomes of the treatment for depression and anxiety were included. Seven of the 17 studies reported multiple groups, of which 5 combined results on multiple treatments in the published study and 2 reported distinct forms of guidance within the same treatment and setting without randomizing patients on an individual level. Of the resulting 30 groups, 8 groups focused on depression, 17 on anxiety, and 5 on both depression and anxiety. We included studies reporting on both depression and anxiety in both the depression and anxiety analyses. Of the included studies, 46.7% (k=14/30, kDep=4, kAnx=10) administered diagnostic interviews, 36.7% (k=11/30; kDep=6/13, kAnx=7/22) self-reports, and 16.7% (k=5/30; kDep=3/13, kAnx=2/22) clinical judgment in their diagnostic process. Of the included studies, 30.0% (k=9/30; kDep=4/13, kAnx=5/22) administered a cutoff criterion for the self-report questionnaires. Ten studies (33.3%, k=10/30; kDep=6/13, kAnx=5/22) included patients who did not meet the criteria for a clinical diagnosis of depression or anxiety. A total of 9.1% of anxiety studies (k=2/22) excluded cases with severe symptom severity; all depression studies allowed patients with severe depression severity to be included. Of the included studies, 40.0% (k=12/30; kDep=4/13, kAnx=9/22) specifically stated that the patients had to be diagnosed with a clinical depression and/or anxiety disorder to follow the iCBT intervention. The rest of the studies did not specify whether the patients had clinical depression and/or anxiety. Of the included studies, 73.0% (k=22/30; kAnx=9/13, kDep=18/22) stated that suicidal ideation or intent was a reason for excluding the patient from the service.

    Table 1. Study characteristics.
    View this table

    iCBT Service–Related Characteristics

    Of the studies, 26.3% (k=5/19) used transdiagnostic interventions, and all others utilized disorder-specific interventions. We did not identify any blended treatments.

    Of the included studies, 31.6% (k=6/19) involved clinical referrals in their service pathway, 26.3% (k=5/19) did not involve referrals, but only included patients through the general community, whereas 42.1% (k=8/19) were recruited in both a community and clinical setting.

    On average, iCBT treatments included 8.00 sessions (SD 2.62; k=26; depression: k=11; mean 8.09, SD 2.84; anxiety: k=19; mean 8.00, SD 2.81).

    With regard to guidance, 46.7% of the studies (k=14/30; kDep=5/13, kAnx=11/22) stated that guidance focused mainly on motivational and 16.7% (k=5/30; kDe=4/13, kAnx=4/22) on administrative aspects. All included studies provided feedback on the content of participants who completed the sessions. Of the studies, 73.3% (k=22/30; kDep=10/13, kAnx=14/22) used asynchronous contact methods for communication between participants and guides, 30.0% (k=9/30; kDep=7/13, kAnx=4/22) used build-in message systems, and 16.7% (k=5/30; kDep=1/13, kAnx=5/22) used emails. Of the studies, 23.3% (k=7/30; kDep=3/13, kAnx=7/22) used synchronous contact via telephone contacts, of which one would also use face-to-face contacts. Of the studies, 16.7% (k=5/30; kDep=1/13, kAnx=4/22) stated that they provided feedback as a reaction following a participant’s action and 30.0% (k=9/30; kDep=4/13, kAnx=6/22) in specific time intervals, weekly or biweekly.

    Of the studies, 23.3% (k=7/30; kDep=4/13, kAnx=6/22) only involved guides not trained in CBT, whereas the other studies included specifically trained professionals, such as psychotherapists, psychiatrists, GPs, or psychologists. Of the studies, 40.0% (k=12/30; kDep=7/13, kAnx=10/22) stated that they provided specific training for the provision of the iCBT intervention to the guides, and 63.3% (k=19/30; kDep=9/13, kAnx=13/22) provided supervision to the guiding participants. A total of 26.7% of the studies (k=8/30; kDep=4/13, kAnx=4/22) reported having provided an iCBT intervention manual. The average reported guidance time was 148.50 min (SD 146.99; k=12; 95% CI 92.87-204.12; depression: k=4, mean 82.44 min (SD 81.14), 95% CI 45.29-119.60; anxiety: k=9; mean 157.60 min, SD 108.10; 95% CI 92.33-222.86). The pooled results are presented in Table 2.

    Nine studies (47.4%; k=19) reported safety measures in cases of suicidality, suicidal ideation, or severe symptom deterioration. They mention monitoring systems (k=2), risk alerts (k=1), or reviewing the participants’ messages (k=2) as ways to identify risk. Procedures were triggered in cases of suicidal ideation or suicidality (k=9), inactivity or lack of progress (k=2), or an increase in symptoms (k=2). Ways to mitigate the risk included contacting the participant via telephone (often in contrast to the usual messaging, k=4), structured risk assessments (k=1), referred to another service (k=6), and the development of a safety plan together with the participant (k=1). Information is depicted in Multimedia Appendix 5 [64-101] and Multimedia Appendix 6 [102-120] on the iCBT service–related characteristics.

    Table 2. Pooled results of iCBT service– and acceptability-related outcomes: guidance time, age, gender, completed sessions, completed components, and deterioration rates.
    View this table

    Risk of Bias of the Included Studies

    The quality of the included studies varied. Of the studies, 67.0% (k=20/30) were rated with a high risk of bias on Researcher Allegiance. Of the studies, 63.0% (k=19/30) did not exclude patients who were participating in other psychotherapeutic treatments (Treatment Inclusion Confounding), and none of the studies reported on the adjustment for confounders in the data analysis. Intention-to-treat data could be extracted from 73.3% of the studies (k=22/30), and none of the studies were preceded by a published study protocol. The risk of bias assessment is depicted in Figure 2.

    Figure 2. Risk of bias assessment.
    View this figure

    iCBT Service Acceptability

    Acceptability data on uptake, participant characteristics across studies, adherence, and participant satisfaction were pooled. All acceptability results are depicted in Multimedia Appendices 6 and 7. The pooled results are presented in Table 2.

    Uptake

    The average proportion of included people based on the number of people screened was 70.2% (k=6/30; 95% CI 8.4%-98.4%; range=0.6%-76.0%), the proportion of starters based on the number of people being screened was 48.0% (k=10; 95% CI 16.9%-80.8%; range=0.3%- 96.2%), and the proportion of starters based on the number of people included was 73.0% (k=7; 95% CI 51.0%-87.6%; range=40.6%- 95.9%).

    Participant Characteristics

    The pooled percentage of female participants was 65.4% (k=23, 95% CI 57.2%-72.8%; depression: k=11, mean 70.1%, 95% CI 55.7%-81.4%; anxiety: k=17, mean 64.3%, 95% CI 56.1%-71.6%). The mean age across studies was 38.30 years (k=29, 95% CI 37.22-39.37; depression: k=12, mean 38.96, 95% CI 37.77-40.15; anxiety: k=21, mean 37.83, 95% CI 36.47-39.20).

    Adherence

    The average percentage of sessions completed was 61.2% (k=14, 95% CI 54.9%-67.5%; depression: k=5, mean 62.6%, 95% CI 61.2%-63.9%; anxiety: k=10, mean 57.3%, 95% CI 56.1%-58.4%). The percentage of participant completing all treatment components was 61.3% (k=26, 95% CI 55.3%-66.9%; depression: k=12, mean 62.8%, 95% CI 55.1%-70.0%; anxiety: k=18, mean 61.7%, 95% CI 53.5%-69.3%).

    Participant Satisfaction

    Of the 17 studies, 10 (58.8%) reported participants’ satisfaction. Participant satisfaction outcomes were reported inconsistently, using varying measures and different reporting forms. Therefore, these data could not be pooled, but the detailed results and the data extracted on patient satisfaction are depicted in Multimedia Appendix 7 [81,121-123]. Within the studies reporting participants’ satisfaction, five studies reported a high and four a very high participants’ satisfaction.

    Effects of iCBT on Symptom Change

    Depression

    Effect sizes for changes in depression severity ranged from 0.66 to 1.88 (Hedges’ g, k=13 studies), with 1 study (7.7%) reporting a moderate and 12 (92.3%) a large effect size.

    The average pre-post effect size of all depression treatments was g=1.18 (95% CI 1.06-1.29), which can be considered a large effect. Heterogeneity was significant and high (I2=95%; 95% CI 94-97; P<.001). The prediction interval is 0.74-1.62, and we can expect that in 95% of all populations, the true effect size will fall within this range.

    The details of these results are shown in Figure 3 and Table 3.

    Figure 3. Standardized Effects of iCBT treatments for depression in routine care. Full references are available in Multimedia Appendix 4. Combined: multiple measures for the main outcome have been combined in the analysis; Dep.: depression treatment; Mixed: mixed depression and anxiety treatment; NS: nonspecialized care; PHQ 8: Patient health Questionnaire – 8 Item version; PHQ 9: Patient Health Questionnaire; Plus: depression treatment for older adults; Spec.: specialized care.
    View this figure
    Table 3. Meta-analytic comparison of anxiety and depression interventions.
    View this table

    In this analysis, the pre-post measurement correlation was set to the actual pre-post correlation of the measure (between 0.36 and 0.78). Sensitivity analysis, with correlations set to 0, 0.75, and 0.99, resulted in comparable effect sizes (gCorr=0=1.24, I2Corr=0=86, 95% CI 78-91; P<.001; gCorr=.75=1.16, I2Corr=.75=96, 95% CI 94-97; P<.001), with gCorr=0.99=0.75 (I2Corr=.99=100, 95% CI 100-100; P<.001) resulting in the smallest effect size.

    Both the visual inspection of the funnel plot and Egger test (P=.90) did not indicate a potential publication bias.

    We found five studies to be outliers, defined as not overlapping with the 95% CI of the pooled estimate. Removing these studies [47-49], and the depression group in the study by Ruwaard et al [50], from the analysis did not result in meaningful changes in effect sizes (g=1.18, 95% CI 1.09-1.26), but reduced heterogeneity (I2=75%; 95% CI 42-86; P<.001). Removing the mixed anxiety and depression studies did not result in a relevant change in effect size (g=1.28; 95% CI 1.13-1.44; I2=97%; 95% CI 95-98; P<.001).

    Anxiety

    For the included anxiety studies (k=20), effect sizes ranged from 0.42 to 1.38 (Hedges’ g), with 1 study (5.0%) reporting a small, 6 (30.0%) a moderate, and 13 (65.0%) a large effect size.

    The average pre-post effect size (Hedges’ g) of all anxiety interventions, including the interventions that targeted both anxiety and depression, was g=0.94 (95% CI 0.83-1.06), which is considered a large effect. Heterogeneity was high (I2=89, 95% CI 84-92; P<.001). The prediction interval is 0.44-1.44, and we can expect that in 95% of all populations, the true effect size will fall within this range. The details of these results are shown in Figure 4 and Table 3.

    In the main analysis described above, the pre-post measurement correlation was set to 0.59. Sensitivity analysis with correlations set to 0, 0.75, and 0.99 resulted in comparable effect sizes (gCorr=0=0.95, I2Corr=0=74, 95% CI 60-83; P<.001; gCorr=.75=0.93, I2Corr=.75=93, 95% CI 91-95; P<.001), with gCorr=0.99=0.70 (I2Corr=.99=99, 95% CI 99-100; P<.001) resulting in the smallest effect size.

    Figure 4. Standardized Effects of iCBT treatments for anxiety in routine care. Marks (2003) is not providing an anxiety measure for the mixed depression and anxiety treatment; therefore, this study has not been included in the analysis. Full references are available in Multimedia Appendix 4. Combined: multiple measures for the main outcome have been combined in the analysis; GAD: generalized anxiety disorder; GP: general practitioner-guided; LSAS: Liebowitz Social Anxiety Scale; NS: nonspecialized care; OCD: obsessive-compulsive disorder; PDSS-SR: Panic Disorder Severity Scale-Self Report; PTSD: posttraumatic stress disorder; Spec.: specialized care; Th.: therapist-guided; YBOCS: Yale-Brown Obsessive Compulsive Scale.
    View this figure

    Both the visual inspection of the funnel plot and Egger test (P=.91) did not indicate a potential publication bias.

    We found five studies to be outliers, as their results did not overlap with the 95% CI of the pooled estimate. Removing studies [51,52] as well as PTSD and panic disorder studies by Ruwaard et al [50] and PTSD study by Titov et al [53] from the analysis did not influence the result significantly (g=0.90; 95% CI 0.81-0.99; I2=77%, 95% CI 62-86; p<.001), but resulted in less, although still high, heterogeneity. Excluding the mixed anxiety and depression studies did not result in a significantly different effect size (g=0.99; 95% CI 0.86-1.12; I2=92%; 95% CI 88-95; P<.001). Neither removing OCD treatments (g=0.93; 95% CI 0.82-1.05; I2=90%; 95% CI 86-93; P<.001), PTSD treatments (g=0.88; 95% CI 0.78-0.98; I2=83%, 95% CI 74-89; P<.001), or both (g=0.87; 95% CI 0.77-0.98; I2=84%; 95% CI 76-90; P<.001) resulted in significantly different effect sizes, although lowering the heterogeneity.

    iCBT Negative Effects

    Multimedia Appendix 7 [81,121-123] comprises the results of the negative effects. Less than half of the studies reported deterioration rates (k=7; 41%), with an average deterioration rate of 2.9% (k=14, 95% CI 1.9%-4.3%; depression: k=5, mean 2.5%, 95% CI 2.2%-2.9%; anxiety: k=9, mean 3.1%, 95% CI 1.6%-5.9%; the forest plot can be retrieved from the corresponding author). No study reported other negative effects, and one study mentioned that there were no adverse outcomes. No studies have reported the predictors of deterioration or other negative effects.

    Subgroup Analysis for iCBT for the Treatment of Depression, Anxiety, or Mixed Depression and/or Anxiety

    Tables 4 and 5 show the results of all examined subgroup analyses. Significant differences between subgroups were found for professional training of coaches, supervision of coaches, and treatment duration for both depression and anxiety studies and for recruitment pathways for depression studies only. Studies evaluating a period of 9 to 13 weeks of treatment duration reported a significant lower effect size (depression: g=1.00, 95% CI 0.95-1.05; I2=0; 95% CI 0-85; anxiety: g=0.83, 95% CI 0.72-0.95; I2=59; 95% CI 9-81) compared with studies with less than 9 (depression: g=1.17, 95% CI 1.01-1.32; I2=95; 95% CI 92-97; anxiety: g=1.16, 95% CI 0.97-1.34; I2=93; 95% CI 61-91) or more than 13 weeks (depression: g=1.37, 95% CI 1.00 to −1.74; I2=97; 95% CI 94-98; anxiety: g=0.98, 95% CI 0.78-1.17; I2=89; 95% CI 78-94). However, the effect sizes within all examined subgroups were high.

    Table 4. Subgroup analyses: depression treatments.
    View this table
    Table 5. Subgroup analyses: anxiety treatments.
    View this table

    Depression studies that recruited in community settings only reported significantly higher effect sizes (g=1.37, 95% CI 1.16-1.59; I2=96; 95% CI 94-98), compared with studies that recruited in clinical or clinical and community settings (g=1.05, 95% CI 0.95-1.14; I2=78; 95% CI 56-89). Across all recruitment pathways, effect sizes were large, but heterogeneity remained high. We did not find this difference in anxiety studies.

    Studies only involving guides, not trained in CBT, showed a significantly lower effect size in depression studies (gNon-professional, Depression=0.92, 95% CI 0.79-1.05; I 2=22; 95% CI 0-88) than all other studies, including specifically trained professionals (gOther,Depression=1.27, 95% CI 1.14-1.40; I2=96; 95% CI 94-97). We did not find this effect in anxiety studies (gNon-professional,Anxiety=0.87, 95% CI 0.17-1.27; I2=88; 95% CI 73-95; gOther,Anxiety=0.96, 95% CI 0.83-1.09; I2=90; 95% CI 85-93).

    Depression studies reporting to having provided supervision to their coaches, trained their professionals, and provided an intervention manual reported a significantly higher effect size (gSupervision=1.27, 95% CI 1.13-1.41; I2=96; 95% CI 94-97; gTraining=1.35, 95% CI 1.20-1.51; I2=95; 95% CI 91-97; gManual=1.47, 95% CI 1.23-1.71; I2=97; 95% CI 95-98) compared with studies not reporting to provide these (gNoSupervision=0.91, 95% CI 0.75-1.08; I2=39; 95% CI 0-79; gNoTraining=0.98, 95% CI 0.94-1.02; I2=7; 95% CI 0-76; gNoManual=1.04, 95% CI 0.95-1.13; I2=75; 95% CI 51-87). For anxiety studies, we found similar effects for the reporting of training and providing an intervention manual (gTraining=1.07, 95% CI 0.89-1.26; I2=90; 95% CI 85-94; gManual=1.30, 95% CI 1.19-1.41; I2=29; 95% CI 0-74 compared with gNoTraining=0.80, 95% CI 0.67-0.93; I2=83; 95% CI 70-88; gNoManual=0.88, 95% CI 0.75-0.94; I2=81; 95% CI 70-88), but not for supervision.

    There were no differences between subgroups regarding all other examined subgroups, both for depression and anxiety studies.

    Subgroup analyses comparing studies rated with high versus low risk indicated that Researcher Allegiance did not have a significant influence on the estimated effect sizes for neither anxiety nor depression studies. The heterogeneity within the studies reporting a low risk of bias on Researcher Allegiance did reveal an I2 of 39 compared with an I2 of 90 for studies reporting a high risk of bias. Moreover, anxiety studies rated as at high risk of Treatment Inclusion Confounding had higher estimated effect sizes. This was not replicated in subgroup analyses of interventions targeting depression. Anxiety studies at high risk of Selection Bias reported significantly lower effect sizes. Similar outcomes were not replicated in the depression trials.

    Meta-Regression Analysis for iCBT for the Treatment of Anxiety, Depression, or Mixed Depression and Anxiety

    Meta-regression analyses indicated that longer treatment duration in depression studies was positively associated with a higher effect (P=.02; β=0.03, R2=0.00). This effect was not found in anxiety studies (P=.94). None of the examined variables, that is, guidance time, number of contacts, number of sessions completed, or the percentage of treatment completers, were significantly associated with the observed effect sizes, neither in depression nor anxiety studies.


    Discussion

    This study aims to examine the acceptability, effects on symptom change, and negative effects of guided iCBT interventions in treating depression and anxiety in routine care. Regarding the uptake of the service, on average, 70.2% of people screened were not offered inclusion, and of those included, 73.0% started the intervention. The vast majority of participants reached were female, with an average age of 38.3 years, and 61.3% of participants completed the interventions as planned. Reported participant satisfaction was high, although inconsistently reported results did not allow us to pool effects. The average professional guidance time per participant was 133.49 min over the treatment duration. With regard to the effects on symptom change, the results indicated large average reductions for both depression (g=1.18; 95% CI 1.06-1.29) and anxiety (g=0.94; 95% CI 0.83-1.062). However, the heterogeneity between studies was high. Nevertheless, all examined effect sizes were at least moderate, indicating the intervention’s potential when delivered under routine care conditions with effects ranging from moderate to large. The average deterioration rates were 3.2% for depression and 3.1% for anxiety. Subgroup analyses indicated a range of iCBT service–related characteristics to be associated with the observed treatment effects.

    Regarding uptake, we found that many participants who were in contact with the iCBT service did not start the intervention. Pretreatment dropout is hard to assess, and, accordingly, reasons for not starting an iCBT intervention after inclusion have not been discussed in the original publications.

    The average age of participants found in this study (mean 38.30) appears to be slightly lower than that reported in RCTs on guided iCBT interventions for the treatment of depression (mean 42.5 [124]) but comparable with reports on the mean age of participants within guided iCBT interventions for the treatment of anxiety [125]. The percentage of females in the routine care study population was higher for depression studies compared with guided iCBT for the treatment of depression [124] and similar to reports on participants in guided iCBT interventions for the treatment of anxiety [125] in experimental settings. As similar distributions between female and male users are reported in face-to-face mental health service utilization [126], this effect might be explained by gender differences in help-seeking behavior than being related to iCBT service–related factors [127] as well as by gender differences in the prevalence of depression and anxiety disorder [128,129]. Future studies should focus on ways to attract men to use iCBT interventions.

    The pooled reported percentage of sessions completed, that is, 62.6% in depression and 57.3% in anxiety studies, was lower than that described in meta-analyses on adherence in RCTs on iCBT interventions. Comparing the adherence to iCBT and face-to-face CBT, van Ballegooijen et al [130] reported that on average, participants completed 80.8% of treatment sessions in the iCBT and 83.9% in the face-to-face intervention [130]. Similarly, the percentage of participants completing the treatment as planned was lower (62.8% for depression and 61.7% for anxiety studies) than reported elsewhere [130,131]. These differences might be due to the assumed adherence-fostering effect of randomized controlled settings versus routine care [132]. However, completion rates were reported inconsistently across studies, applying different criteria such as study or treatment completers, including several definitions of treatment completions. To facilitate comparability, literature on iCBT completion should settle on one reporting standard. Further investigation of factors promoting the acceptance of iCBT interventions, also when reporting on effectiveness results in routine care, may lead to a deeper understanding that might foster intervention development and upscaling.

    Results on the effectiveness of iCBT (gDepression=1.18, 95% CI 1.06-1.29 and gAnxiety=0.94, 95% CI 0.83-1.062) confirm findings of recently published systematic reviews and meta-analyses on RCTs of iCBT for depression and anxiety. Königbauer et al [12] found medium to large pre-post within-group effects ranging between −0.64 and −2.24 for interventions treating clinical depression [12]. To our knowledge, no recent meta-analysis has reported on pre-post effect sizes of studies targeting guided iCBT interventions for the treatment of anxiety. On an individual study level, pre-post effects in randomized trials ranged from 0.54 to 2.40 (please see Multimedia Appendix 8 for references) [133-162] compared with 0.66 to 1.88 in depression and 0.42 to 1.38 (Hedges’ g) in anxiety within this analysis.

    With regard to randomized pragmatic trials conducted under routine care conditions, Andrews et al [15] examined a sample of 64 papers reporting results of RCTs on the effectiveness of iCBT for the treatment of depression, panic disorder, generalized anxiety disorder, and social phobia in comparison with control groups in routine practice. This review study reported effect sizes for depression, panic disorder, generalized anxiety disorder, and social phobia ranging from g=0.67 to 1.31 [15]. The same study identified eight papers investigating the effectiveness of iCBT, reporting an average effect size of g=1.07 across the treatment of depression, panic disorder, generalized anxiety disorder, and social phobia [15]. The between-group effects were moderate to large (g=0.72; 95% CI 0.60-0.83; P<.001; of =53, 95% CI 31-66) in the most recent meta-analysis of iCBT treatments for anxiety compared with control conditions in reducing symptoms of anxiety in an adult population [13]. Additionally, the results of this study are in line with meta-analytic findings on face-to-face CBT treatments implemented in routine care with pre-post effect size found in randomized trials ranging from d=0.69 to 2.28 for depression [28] and g=0.73 to 2.59 for anxiety treatments [163].

    The results of deterioration rates (3.2% in depression and 3.1% in anxiety studies) were slightly lower, but within the 95% CI of findings based on RCTs for internet-based guided self-help interventions (3.36%) for depression [164] and anxiety (5.8% [165]), and also comparable with deterioration rates in face-to-face psychotherapy for depression [166]. Criteria defining deterioration varied between studies, and unfortunately, neither were reports on other negative effects included in most primary studies nor reported any study predictors of deterioration. This seems of utmost importance to identify those individuals that should potentially be referred to other mental health services. Their investigation is of specific importance within naturalistic study designs and under routine care conditions [164,165,167].

    Most evaluated iCBT services for depression (69.2%) excluded severe cases and individuals with suicidal ideation (k=9/13) at baseline. However, a large-scale study showed that iCBT services can also result in positive effects on suicidality, reducing the prevalence of suicidal ideation from 50% at baseline to 27% after treatment [168]. In addition, a recent individual patient data meta-analysis on RCTs indicated that guided iCBT also resulted in clinically meaningful results in individuals with severe depression symptomatology [124]. Given that many individuals applying to iCBT services either do not have access to other immediate care or are not willing to utilize alternative treatment services, future studies should explore the balance between potential risk and benefits of opening up those services to populations showing elevated suicidal ideation. In such cases, it seems of utmost importance to monitor potential upcoming crises using standard operating procedures involving trained clinicians and to evaluate treatment success at the end of the service. In case of nonresponse, individuals should be motivated and guided to utilize other mental health care services, if available. Such standardized crisis procedures were only reported to be employed by less than half of the studies included in this review. iCBT services in routine care might profit from clear pathways of referral to other services in cases of nonresponse and symptom deterioration. Furthermore, future research should facilitate our understanding of the effects of routine outcome monitoring in routinely applied iCBT [169], as this monitoring could help evaluate participants’ progress throughout the course of treatment, using standardized outcome measures to elicit clients as part of a measurement-based care delivery approach in routine mental and behavioral health care [170,171].

    The finding that treatment outcomes of depression interventions were greater when recruitment was carried out using an open recruitment strategy in a community setting compared with when recruited in a clinical setting is in line with the findings of Romijn et al [13] with regard to randomized pragmatic studies on anxiety disorder treatments. However, in our study, this interaction was only found for depression and could not be confirmed for anxiety disorders. One potential explanation for the difference in effects might be differences in the characteristics of the included patients. There is evidence that iCBT recruiting via open recruitment strategies, such as through web-based channels, might only reach a specific population that is different from those seeking help in a clinical setting [19]. It is often argued that internet interventions might reach individuals that would otherwise not seek treatment or only at a later time point. Given that, for example, the chronicity of depression is associated with worse treatment outcomes [172], the difference in effect might be explained by reaching a population with lower chronicity. However, such an assumption needs to be confirmed in future studies.

    Further subgroup analyses indicated that iCBT services for the treatment of depression utilize trained professionals (psychotherapists and psychiatrists) to result in larger pre-post changes compared with iCBT services that used only nonprofessionals not trained in CBT (psychologists without specialized CBT training, nurses, GPs, counselors, coaches, and lived experience coordinators). However, we did not find this effect in the anxiety studies. Moreover, effects in the subgroup of depression studies involving nonprofessionals were large, indicating the potential to deliver iCBT services, for example, in contexts when there might be a shortage of trained clinicians. In cases where nonprofessionals deliver guidance in iCBT services, supervision by trained clinicians, including the availability of professionals for crisis intervention, seems warranted. Further subgroup analyses also indicated that providing supervision to coaches is also associated with higher average treatment effects for depression, but not for anxiety studies. Furthermore, training the professional and providing an intervention manual is positively related to the interventions’ effectiveness. This result must be interpreted with caution as we coded all studies not mentioning supervision, training, or manual provision in their publication as not providing these components. Furthermore, these components do not inform us about actual treatment fidelity. Further research should focus on the effects on treatment outcomes of providing supervision, training, and intervention manuals to professionals working with iCBT interventions in routine care as well as the assessment of treatment fidelity.

    Moreover, we did not find a difference in effects on mean symptom change between iCBT services who applied diagnostic interviews for patient allocation versus those that used self-reports only. This is in line with meta-analytic findings from RCTs on guided digital interventions for depression [124] and with studies directly comparing the effectiveness of iCBT services when treatment allocation was based on an automatic web-based assessment versus clinician assessment [173]. This indicates that such services can be used in contexts when implementing services with initial clinician assessment is not possible, without affecting average treatment success. However, it must be noted that although results might not indicate differences on the group level, it might be the case that using web-based assessments only, without a clinical assessment, will overlook relevant diagnostic information that requires immediate attention, such as suicidal risk or an underlying treatment need for comorbid disorders such as PTSD on an individual level.

    The strengths of this meta-analysis include the exclusive focus on evaluating iCBT interventions for their acceptability and clinical outcomes under real-world conditions. Unlike previous systematic reviews that mixed efficacy with effectiveness trials, in this review, we focused only on studies conducted in regular care settings. This is important as we strive to report routine care results free from biases possibly being introduced within efficacy studies such as stricter application of protocolized procedures, eligibility criteria, and randomization [19-22]. Moreover, we presented an overview of implementation indicators existing in the included studies that can be used to gain a better understanding of how iCBT can be adopted by regular care services. Nevertheless, the findings of this study should be interpreted with caution due to several limitations.

    First, the heterogeneity in our sample was high and significant, illustrating a great variation in the results of the included studies. Thus, we cannot draw firm conclusions regarding the average effect of iCBT in routine care. Moreover, within-group effect sizes do not depict an optimal estimator for the treatment effect because they are not independent of each other and do not account for recovery occurring independent of the treatment, thereby leading to an overestimation of the treatment effect [42]. However, in comparison with and on the basis of the reported efficacy of iCBT interventions established in RCTs, they depict the best available indicator of the effects of iCBT solutions in a routine care environment. Furthermore, we found that treatment duration had a significant influence on treatment effects. This result also supports the hypothesis that findings on pre-post changes in symptom severity might have been influenced by spontaneous or unexplained recovery, which is a common factor in depression [174]. However, our main results are in line with within-group effect sizes found in RCTs, where spontaneous recovery also occurs, and we, therefore, conclude that our effects can be considered substantial. Although heterogeneity was not explained by any other of the examined subgroups, several assumptions can be made regarding its sources. One other explanation for the high heterogeneity might be the influence of contextual factors of observational studies, such as sampling methods, participant characteristics, within-group effect sizes, and differences between the studies in reporting outcomes. It can be hypothesized that a greater harmonization regarding the conduct and reporting of effectiveness studies in routine care could lead to greater comparability of the studies’ results. Another reason for the observed heterogeneity might be the different contexts of regular care facilities across different countries. There is great variability in the degree of e-mental health penetration in different countries. For instance, Australia is considered one of the frontrunners in the e-mental health field, whereas Norway adopted these interventions very recently [175]. Thus, professionals might differ in the way they interact with e-mental health around the world. Finally, the interventions might differ in the way that they have been developed. These results also imply the importance of establishing a firm evidence base for individual iCBT interventions before their larger upscale.

    Second, firm conclusions on treatment effects might be biased by studies that also included participants who could also participate in other psychotherapeutic treatments. Meanwhile, the data do not allow conclusions on the percentage of participants receiving additional treatment and represents the routine practice. Additionally, no study has reported adjusting for confounders such as baseline symptom severity, treatment fidelity (provision and use), or changes in the treatment over the course of the studies, which should be considered in future reports on the effects of iCBT in routine care.

    Future studies should add to the body of literature on iCBT interventions examined under routine care conditions. Additionally, these studies should not solely focus on the effectiveness of the interventions, but if possible, it would be helpful if they also reported on specific service-, implementation-, and context-related outcomes. One way of achieving this might be through taxonomy and guidelines for the reporting of iCBT effectiveness, implementation, and context outcomes in routine care. In contrast to standards of reporting RCTs, no such international standards exist when it comes to reporting nonrandomized intervention studies. Proctor et al [176] suggested a list of outcomes for implementation-related research, and Hermes et al [177] recently made suggestions on how to build upon these ideas to establish a measurement system for the implementation of behavioral intervention technologies. Moreover, such research should always be discussed and evaluated in the light of the quality criteria established to help all involved stakeholders, patients, practitioners, and decision makers at the local and policy level to identify not only effective but also safe interventions [178].

    In conclusion, this study provides further evidence supporting the acceptability and effectiveness of guided iCBT for the treatment of depression and anxiety when implemented in routine care, whereas results on negative effects are less clear. Guided iCBT may be an effective way of overcoming barriers to treatment provision. It may substantially increase the coverage of usual care services and offer an innovative treatment format for the treatment of depression and anxiety.

    Acknowledgments

    The authors wish to acknowledge Silja Schenk’s assistance in managing the literature search in an early stage of the project. The authors also want to thank Dr Robin Kok for providing his knowledge on his website.

    Authors' Contributions

    AE and DE initiated and conceptualized the study and drafted the manuscript with initial feedback from all authors. AE, CV, and DE acquired and managed the data. AE, CV, and EK conducted the data analysis, with critical feedback from DE and PC. All authors revised and edited the manuscript, provided final approval for this version, and agreed to be accountable for this work.

    Conflicts of Interest

    Associate Prof DE reports to have received consultancy fees or served in the scientific advisory board for several companies such as Minddistrict, Sanofi, Lantern, and Schön Kliniken; German health insurance companies (BARMER and Techniker Krankenkasse); and chambers of psychotherapists. Dr DE is one of the stakeholders of the Institute for Health Training Online (GET.ON), which aims to implement scientific findings related to digital health interventions into routine care. AE is employed by the Institute for Health Training Online (GET.ON) as a research coordinator. Prof. NT is funded by the Australian Government to develop and provide a free national online and telephone-delivered treatment service. Prof. HB served in the e-mental health–associated scientific advisory boards, e-mental health interest groups, and task forces. All other authors do not report any conflicts of interest.

    Multimedia Appendix 1

    Prisma Checklist.

    DOCX File , 18 KB

    Multimedia Appendix 2

    Search strategy.

    DOCX File , 13 KB

    Multimedia Appendix 3

    Risk of Bias Assessment Definition.

    DOCX File , 16 KB

    Multimedia Appendix 4

    References of included studies.

    DOCX File , 16 KB

    Multimedia Appendix 5

    E and F - iCBT service-related characteristics.

    DOCX File , 67 KB

    Multimedia Appendix 6

    Uptake and participant characteristics.

    DOCX File , 53 KB

    Multimedia Appendix 7

    Acceptability - Participant satisfaction and negative effects.

    DOCX File , 46 KB

    Multimedia Appendix 8

    References for studies targeting guided iCBT interventions for the treatment of anxiety reporting on pre-post effect sizes.

    DOCX File , 17 KB

    References

    1. Kessler RC, Bromet EJ. The epidemiology of depression across cultures. Annu Rev Public Health 2013;34:119-138 [FREE Full text] [CrossRef] [Medline]
    2. Kessler RC, Berglund P, Demler O, Jin R, Koretz D, Merikangas KR, National Comorbidity Survey Replication. The epidemiology of major depressive disorder: results from the national comorbidity survey replication (NCS-R). J Am Med Assoc 2003 Jun 18;289(23):3095-3105. [CrossRef] [Medline]
    3. Donohue JM, Pincus HA. Reducing the societal burden of depression: a review of economic costs, quality of care and effects of treatment. Pharmacoeconomics 2007;25(1):7-24. [CrossRef] [Medline]
    4. Whiteford H, Degenhardt L, Rehm J, Baxter A, Ferrari A, Erskine H, et al. Global burden of disease attributable to mental and substance use disorders: findings from the global burden of disease study 2010. Lancet 2013 Nov 9;382(9904):1575-1586. [CrossRef] [Medline]
    5. Cuijpers P, Karyotaki E, de Wit L, Ebert DD. The effects of fifteen evidence-supported therapies for adult depression: a meta-analytic review. Psychother Res 2020 Mar;30(3):279-293. [CrossRef] [Medline]
    6. Wittchen HU, Jacobi F, Rehm J, Gustavsson A, Svensson M, Jönsson B, et al. The size and burden of mental disorders and other disorders of the brain in Europe 2010. Eur Neuropsychopharmacol 2011 Sep;21(9):655-679. [CrossRef] [Medline]
    7. Mack S, Jacobi F, Gerschler A, Strehle J, Höfler M, Busch MA, et al. Self-reported utilization of mental health services in the adult German population--evidence for unmet needs? Results of the DEGS1-mental health module (DEGS1-MH). Int J Methods Psychiatr Res 2014 Sep;23(3):289-303 [FREE Full text] [CrossRef] [Medline]
    8. Ebert DD, Nobis S, Lehr D, Baumeister H, Riper H, Auerbach RP, et al. The 6-month effectiveness of Internet-based guided self-help for depression in adults with type 1 and 2 diabetes mellitus. Diabet Med 2017 Jan;34(1):99-107. [CrossRef] [Medline]
    9. Ebert D, van Daele T, Nordgreen T, Karekla M, Compare A, Zarbo C, et al. Internet- and mobile-based psychological interventions: applications, efficacy, and potential for improving mental health. Eur Psychol 2018 May;23(2):167-187. [CrossRef]
    10. Ebert DD, Cuijpers P, Muñoz RF, Baumeister H. Prevention of mental health disorders using internet- and mobile-based interventions: a narrative review and recommendations for future research. Front Psychiatry 2017;8:116 [FREE Full text] [CrossRef] [Medline]
    11. Sander L, Rausch L, Baumeister H. Prevention of mental health disorders using internet- and mobile-based interventions: a narrative review and recommendations for future research and effectiveness of internet-based interventions for the prevention of mental disorders: a systematic review and meta-analysis. JMIR Ment Health 2016 Aug 17;3(3):e38 [FREE Full text] [CrossRef] [Medline]
    12. Josephine K, Josefine L, Philipp D, David E, Harald B. Internet- and mobile-based depression interventions for people with diagnosed depression: a systematic review and meta-analysis. J Affect Disord 2017 Dec 1;223:28-40. [CrossRef] [Medline]
    13. Romijn G, Batelaan N, Kok R, Koning J, van Balkom A, Titov N, et al. Internet-delivered cognitive behavioral therapy for anxiety disorders in open community versus clinical service recruitment: meta-analysis. J Med Internet Res 2019 Apr 17;21(4):e11706 [FREE Full text] [CrossRef] [Medline]
    14. Andersson G, Carlbring P, Titov N, Lindefors N. Internet interventions for adults with anxiety and mood disorders: a narrative umbrella review of recent meta-analyses. Can J Psychiatry 2019 Jul;64(7):465-470 [FREE Full text] [CrossRef] [Medline]
    15. Andrews G, Basu A, Cuijpers P, Craske MG, McEvoy P, English CL, et al. Computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: an updated meta-analysis. J Anxiety Disord 2018 Apr;55:70-78 [FREE Full text] [CrossRef] [Medline]
    16. Carlbring P, Andersson G, Cuijpers P, Riper H, Hedman-Lagerlöf E. Internet-based vs face-to-face cognitive behavior therapy for psychiatric and somatic disorders: an updated systematic review and meta-analysis. Cogn Behav Ther 2018 Jan;47(1):1-18. [CrossRef] [Medline]
    17. Cuijpers P, Noma H, Karyotaki E, Cipriani A, Furukawa T. Effectiveness and acceptability of cognitive behavior therapy delivery formats in adults with depression: a network meta-analysis. JAMA Psychiatry 2019 Jul 1;76(7):700-707 [FREE Full text] [CrossRef] [Medline]
    18. Wells K. Treatment research at the crossroads: the scientific interface of clinical trials and effectiveness research. Am J Psychiatry 1999 Jan;156(1):5-10. [CrossRef] [Medline]
    19. Rothwell PM. External validity of randomised controlled trials: 'to whom do the results of this trial apply?'. Lancet 2005;365(9453):82-93. [CrossRef] [Medline]
    20. Singal AG, Higgins PD, Waljee AK. A primer on effectiveness and efficacy trials. Clin Transl Gastroenterol 2014 Jan 2;5:e45 [FREE Full text] [CrossRef] [Medline]
    21. Ebert DD, Baumeister H. Internet-based self-help interventions for depression in routine care. JAMA Psychiatry 2017 Aug 1;74(8):852-853. [CrossRef] [Medline]
    22. van der Lem R, van der Wee NJ, van Veen T, Zitman FG. Efficacy versus effectiveness: a direct comparison of the outcome of treatment for mild to moderate depression in randomized controlled trials and daily practice. Psychother Psychosom 2012;81(4):226-234. [CrossRef] [Medline]
    23. Vlahakes GJ. The value of phase 4 clinical testing. N Engl J Med 2006 Jan 26;354(4):413-415 [FREE Full text] [CrossRef]
    24. Hill T. Conducting phase IV clinical studies: a moral imperative? Ecancermedicalscience 2012;6:276. [CrossRef]
    25. Andersson G, Hedman E. Effectiveness of guided internet-based cognitive behavior therapy in regular clinical settings. Verhaltenstherapie 2013;23(3):140-148. [CrossRef] [Medline]
    26. Moher D, Liberati A, Tetzlaff J, Altman DG, PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. PLoS Med 2009 Jul 21;6(7):e1000097 [FREE Full text] [CrossRef] [Medline]
    27. Erbe D, Eichert H, Riper H, Ebert DD. Blending face-to-face and internet-based interventions for the treatment of mental disorders in adults: systematic review. J Med Internet Res 2017 Sep 15;19(9):e306 [FREE Full text] [CrossRef] [Medline]
    28. Hans E, Hiller W. Effectiveness of and dropout from outpatient cognitive behavioral therapy for adult unipolar depression: a meta-analysis of nonrandomized effectiveness studies. J Consult Clin Psychol 2013 Feb;81(1):75-88. [CrossRef] [Medline]
    29. Zarski A, Lehr D, Berking M, Riper H, Cuijpers P, Ebert DD. Adherence to internet-based mobile-supported stress management: a pooled analysis of individual participant data from three randomized controlled trials. J Med Internet Res 2016 Jun 29;18(6):e146 [FREE Full text] [CrossRef] [Medline]
    30. Glossary. National Institute for Health Research.   URL: https://www.nihr.ac.uk/glossary?letter=F&postcategory=-1 [accessed 2020-06-24]
    31. Arain M, Campbell MJ, Cooper CL, Lancaster GA. What is a pilot or feasibility study? A review of current practice and editorial policy. BMC Med Res Methodol 2010 Jul 16;10:67 [FREE Full text] [CrossRef] [Medline]
    32. Higgins JP, Green S. Cochrane Handbook for Systematic Reviews of Interventions. New York, USA: Wiley; 2006.
    33. Sterne JA, Hernán MA, Reeves BC, Savović J, Berkman ND, Viswanathan M, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. Br Med J 2016 Oct 12;355:i4919 [FREE Full text] [CrossRef] [Medline]
    34. Higgins JP, Altman DG. Assessing risk of bias in included studies. In: Cochrane Handbook for Systematic Reviews of Interventions. Chichester, UK: John Wiley & Sons; 2008.
    35. Leykin Y, DeRubeis RJ. Allegiance in psychotherapy outcome research: separating association from bias. Clin Psychol (New York) 2009;16(1):54-65. [CrossRef]
    36. Munder T, Brütsch O, Leonhart R, Gerger H, Barth J. Researcher allegiance in psychotherapy outcome research: an overview of reviews. Clin Psychol Rev 2013 Jun;33(4):501-511. [CrossRef] [Medline]
    37. Hedges L, Olkin I. Statistical Methods Meta-Analysis. New York, USA: Academic Press; 2014.
    38. Cohen J. Statistical Power Analysis for the Behavioral Sciences. New York, USA: Behavioral Sciences; 1988.
    39. R Core Team. R: A Language and Environment for Statistical Computing. Vienna, Austria: R Core Team; 2019.
    40. Balduzzi S, Rücker G, Schwarzer G. How to perform a meta-analysis with R: a practical tutorial. Evid Based Ment Health 2019 Nov;22(4):153-160. [CrossRef] [Medline]
    41. Balk EM, Earley A, Patel K, Trikalinos TA, Dahabreh IJ. Empirical assessment of within-arm correlation imputation in trials of continuous outcomes. Methods Res Reports 2012;12:EHC141. [Medline]
    42. Cuijpers P, Weitz E, Cristea IA, Twisk J. Pre-post effect sizes should be avoided in meta-analyses. Epidemiol Psychiatr Sci 2017 Aug;26(4):364-368 [FREE Full text] [CrossRef] [Medline]
    43. Borenstein M, Higgins J, Hedges LV, Rothstein HR. Basics of meta-analysis: I is not an absolute measure of heterogeneity. Res Synth Methods 2017 Mar;8(1):5-18. [CrossRef] [Medline]
    44. Sterne JA, Sutton AJ, Ioannidis JP, Terrin N, Jones DR, Lau J, et al. Recommendations for examining and interpreting funnel plot asymmetry in meta-analyses of randomised controlled trials. Br Med J 2011 Jul 22;343:d4002. [CrossRef] [Medline]
    45. Sterne J, Egger M. Regression methods to detect publication and other bias in meta-analysis. In: Publication Bias in Meta-Analysis: Prevention, Assessment and Adjustments. New York, USA: John Wiley & Sons; 2006.
    46. Duval S, Tweedie R. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 2000 Jun;56(2):455-463. [CrossRef] [Medline]
    47. Gellatly J, Chisnall L, Seccombe N, Ragan K, Lidbetter N, Cavanagh K. @Home etherapy service for people with common mental health problems: an evaluation. Behav Cogn Psychother 2018 Jan;46(1):115-120. [CrossRef] [Medline]
    48. Hedman E, Ljótsson B, Kaldo V, Hesser H, El Alaoui S, Kraepelien M, et al. Effectiveness of internet-based cognitive behaviour therapy for depression in routine psychiatric care. J Affect Disord 2014 Feb;155:49-58. [CrossRef] [Medline]
    49. Marks IM, Mataix-Cols D, Kenwright M, Cameron R, Hirsch S, Gega L. Pragmatic evaluation of computer-aided self-help for anxiety and depression. Br J Psychiatry 2003 Jul;183:57-65. [CrossRef] [Medline]
    50. Ruwaard J, Lange A, Schrieken B, Dolan CV, Emmelkamp P. The effectiveness of online cognitive behavioral treatment in routine clinical practice. PLoS One 2012;7(7):e40089 [FREE Full text] [CrossRef] [Medline]
    51. Shandley K, Austin DW, Klein B, Pier C, Schattner P, Pierce D, et al. Therapist-assisted, Internet-based treatment for panic disorder: can general practitioners achieve comparable patient outcomes to psychologists? J Med Internet Res 2008 May 19;10(2):e14 [FREE Full text] [CrossRef] [Medline]
    52. Yu JS, Szigethy E, Wallace M, Solano F, Oser M. Implementation of a guided, digital cognitive behavioral program for anxiety in primary care: preliminary findings of engagement and effectiveness. Telemed J E Health 2018 Nov;24(11):870-878. [CrossRef] [Medline]
    53. Titov N, Dear B, Staples L, Bennett-Levy J, Klein B, Rapee R, et al. The first 30 months of the mindspot clinic: evaluation of a national e-mental health service against project objectives. Aust N Z J Psychiatry 2017 Dec;51(12):1227-1239. [CrossRef] [Medline]
    54. Morrison C, Walker G, Ruggeri K, Hacker Hughes J. An implementation pilot of the MindBalance web-based intervention for depression in three IAPT services. Cogn Behav Therap 2014 Nov 14;7:e15. [CrossRef]
    55. Aydos L, Titov N, Andrews G. Shyness 5: the clinical effectiveness of internet-based clinician-assisted treatment of social phobia. Australas Psychiatry 2009 Dec;17(6):488-492. [CrossRef] [Medline]
    56. El Alaoui S, Hedman E, Kaldo V, Hesser H, Kraepelien M, Andersson E, et al. Effectiveness of internet-based cognitive-behavior therapy for social anxiety disorder in clinical psychiatry. J Consult Clin Psychol 2015 Oct;83(5):902-914. [CrossRef] [Medline]
    57. Gellatly J, Chisnall L, Seccombe N, Ragan K, Lidbetter N, Cavanagh K. Home etherapy service for people with common mental health problems: an evaluation. Behav Cogn Psychother 2018 Jan;46(1):115-120. [CrossRef] [Medline]
    58. Hadjistavropoulos H, Nugent M, Alberts N, Staples L, Dear B, Titov N. Transdiagnostic internet-delivered cognitive behaviour therapy in Canada: an open trial comparing results of a specialized online clinic and nonspecialized community clinics. J Anxiety Disord 2016 Aug;42:19-29 [FREE Full text] [CrossRef] [Medline]
    59. Hadjistavropoulos H, Pugh N, Nugent M, Hesser H, Andersson G, Ivanov M, et al. Therapist-assisted internet-delivered cognitive behavior therapy for depression and anxiety: translating evidence into clinical practice. J Anxiety Disord 2014 Dec;28(8):884-893 [FREE Full text] [CrossRef] [Medline]
    60. Hedman E, Ljótsson B, Rück C, Bergström J, Andersson G, Kaldo V, et al. Effectiveness of internet-based cognitive behaviour therapy for panic disorder in routine psychiatric care. Acta Psychiatr Scand 2013 Dec;128(6):457-467. [CrossRef] [Medline]
    61. Mathiasen K, Riper H, Andersen TE, Roessler KK. Guided internet-based cognitive behavioral therapy for adult depression and anxiety in routine secondary care: observational study. J Med Internet Res 2018 Nov 28;20(11):e10927 [FREE Full text] [CrossRef] [Medline]
    62. Nordgreen T, Gjestad R, Andersson G, Carlbring P, Havik O. The effectiveness of guided internet-based cognitive behavioral therapy for social anxiety disorder in a routine care setting. Internet Interv 2018 Sep;13:24-29 [FREE Full text] [CrossRef] [Medline]
    63. Nordgreen T, Gjestad R, Andersson G, Carlbring P, Havik OE. The implementation of guided Internet-based cognitive behaviour therapy for panic disorder in a routine-care setting: effectiveness and implementation efforts. Cogn Behav Ther 2018 Jan;47(1):62-75. [CrossRef] [Medline]
    64. Titov N, Andrews G, Choi I, Schwencke G, Mahoney A. Shyness 3: randomized controlled trial of guided versus unguided internet-based CBT for social phobia. Aust N Z J Psychiatry 2008 Dec;42(12):1030-1040. [CrossRef] [Medline]
    65. Titov N, Andrews G, Schwencke G. Shyness 2: treating social phobia online: replication and extension. Aust N Z J Psychiatry 2008 Jul;42(7):595-605. [CrossRef] [Medline]
    66. Titov N, Andrews G, Schwencke G, Drobny J, Einstein D. Shyness 1: distance treatment of social phobia over the Internet. Aust N Z J Psychiatry 2008 Jul;42(7):585-594. [CrossRef] [Medline]
    67. Furmark T, Carlbring P, Hedman E, Sonnenstein A, Clevberger P, Bohman B, et al. Guided and unguided self-help for social anxiety disorder: randomised controlled trial. Br J Psychiatry 2009 Nov;195(5):440-447. [CrossRef] [Medline]
    68. Andersson G, Carlbring P, Holmström A, Sparthan E, Furmark T, Nilsson-Ihrfelt E, et al. Internet-based self-help with therapist feedback and in vivo group exposure for social phobia: a randomized controlled trial. J Consult Clin Psychol 2006 Aug;74(4):677-686. [CrossRef] [Medline]
    69. Buntrock C, Berking M, Smit F, Lehr D, Nobis S, Riper H, et al. Preventing depression in adults with subthreshold depression: health-economic evaluation alongside a pragmatic randomized controlled trial of a web-based intervention. J Med Internet Res 2017 Jan 4;19(1):e5 [FREE Full text] [CrossRef] [Medline]
    70. Buntrock C, Ebert DD, Lehr D, Smit F, Riper H, Berking M, et al. Effect of a web-based guided self-help intervention for prevention of major depression in adults with subthreshold depression. Obstet Gynecol Surv 2016 May 3;71(9):526-527. [CrossRef]
    71. Nobis S, Lehr D, Ebert DD, Baumeister H, Snoek F, Riper H, et al. Efficacy of a web-based intervention with mobile phone support in treating depressive symptoms in adults with type 1 and type 2 diabetes: a randomized controlled trial. Diabetes Care 2015 May;38(5):776-783. [CrossRef] [Medline]
    72. Nobis S, Ebert DD, Lehr D, Smit F, Buntrock C, Berking M, et al. Web-based intervention for depressive symptoms in adults with types 1 and 2 diabetes mellitus: a health economic evaluation. Br J Psychiatry 2018 Apr;212(4):199-206. [CrossRef] [Medline]
    73. Schlicker S, Weisel KK, Buntrock C, Berking M, Nobis S, Lehr D, et al. Do nonsuicidal severely depressed individuals with diabetes profit from internet-based guided self-help? Secondary analyses of a pragmatic randomized trial. J Diabetes Res 2019;2019:2634094 [FREE Full text] [CrossRef] [Medline]
    74. Ebert DD, Lehr D, Heber E, Riper H, Cuijpers P, Berking M. Internet- and mobile-based stress management for employees with adherence-focused guidance: efficacy and mechanism of change. Scand J Work Environ Health 2016 Sep 1;42(5):382-394 [FREE Full text] [CrossRef] [Medline]
    75. McClay CA, Collins K, Matthews L, Haig C, McConnachie A, Morrison J, et al. A community-based pilot randomised controlled study of life skills classes for individuals with low mood and depression. BMC Psychiatry 2015 Feb 6;15:17 [FREE Full text] [CrossRef] [Medline]
    76. Richards D, Timulak L, O'Brien E, Hayes C, Vigano N, Sharry J, et al. A randomized controlled trial of an internet-delivered treatment: its potential as a low-intensity community intervention for adults with symptoms of depression. Behav Res Ther 2015 Dec;75:20-31. [CrossRef] [Medline]
    77. Littlewood E, Duarte A, Hewitt C, Knowles S, Palmer S, Walker S, REEACT Team. A randomised controlled trial of computerised cognitive behaviour therapy for the treatment of depression in primary care: the randomised evaluation of the effectiveness and acceptability of computerised therapy (REEACT) trial. Health Technol Assess 2015 Dec;19(101):viii, xxi-viii, 171 [FREE Full text] [CrossRef] [Medline]
    78. Proudfoot J, Goldberg D, Mann A, Everitt B, Marks I, Gray J. Computerized, interactive, multimedia cognitive-behavioural program for anxiety and depression in general practice. Psychol Med 2003 Feb;33(2):217-227. [CrossRef] [Medline]
    79. Dear BF, Staples LG, Terides MD, Karin E, Zou J, Johnston L, et al. Transdiagnostic versus disorder-specific and clinician-guided versus self-guided internet-delivered treatment for generalized anxiety disorder and comorbid disorders: a randomized controlled trial. J Anxiety Disord 2015 Dec;36:63-77 [FREE Full text] [CrossRef] [Medline]
    80. Titov N, Dear BF, Johnston L, Lorian C, Zou J, Wootton B, et al. Improving adherence and clinical outcomes in self-guided internet treatment for anxiety and depression: randomised controlled trial. PLoS One 2013;8(7):e62873 [FREE Full text] [CrossRef] [Medline]
    81. Titov N, Dear B, Staples L, Terides M, Karin E, Sheehan J, et al. Disorder-specific versus transdiagnostic and clinician-guided versus self-guided treatment for major depressive disorder and comorbid anxiety disorders: a randomized controlled trial. J Anxiety Disord 2015 Oct;35:88-102 [FREE Full text] [CrossRef] [Medline]
    82. Andersson G, Bergström J, Holländare F, Carlbring P, Kaldo V, Ekselius L. Internet-based self-help for depression: randomised controlled trial. Br J Psychiatry 2005 Nov;187:456-461. [CrossRef] [Medline]
    83. Vernmark K, Lenndin J, Bjärehed J, Carlsson M, Karlsson J, Oberg J, et al. Internet administered guided self-help versus individualized e-mail therapy: a randomized trial of two versions of CBT for major depression. Behav Res Ther 2010 May;48(5):368-376. [CrossRef] [Medline]
    84. Carlbring P, Bohman S, Brunt S, Buhrman M, Westling BE, Ekselius L, et al. Remote treatment of panic disorder: a randomized trial of internet-based cognitive behavior therapy supplemented with telephone calls. Am J Psychiatry 2006 Dec;163(12):2119-2125. [CrossRef] [Medline]
    85. Carlbring P, Westling BE, Ljungstrand P, Ekselius L, Andersson G. Treatment of panic disorder via the internet: a randomized trial of a self-help program. Behav Ther 2001;32(4):751-764. [CrossRef]
    86. Marks IM, Kenwright M, McDonough M, Whittaker M, Mataix-Cols D. Saving clinicians' time by delegating routine aspects of therapy to a computer: a randomized controlled trial in phobia/panic disorder. Psychol Med 2004 Jan;34(1):9-17. [CrossRef] [Medline]
    87. Mathiasen K, Riper H, Ehlers LH, Valentin JB, Rosenberg NK. Internet-based CBT for social phobia and panic disorder in a specialised anxiety clinic in routine care: results of a pilot randomised controlled trial. Internet Interv 2016 May;4:92-98 [FREE Full text] [CrossRef] [Medline]
    88. Schneider AJ, Mataix-Cols D, Marks IM, Bachofen M. Internet-guided self-help with or without exposure therapy for phobic and panic disorders. Psychother Psychosom 2005;74(3):154-164. [CrossRef] [Medline]
    89. Greist JH, Marks IM, Baer L, Kobak KA, Wenzel KW, Hirsch MJ, et al. Behavior therapy for obsessive-compulsive disorder guided by a computer or by a clinician compared with relaxation as a control. J Clin Psychiatry 2002 Feb;63(2):138-145. [CrossRef] [Medline]
    90. Nordgreen T, Haug T, Öst LG, Andersson G, Carlbring P, Kvale G, et al. Stepped care versus direct face-to-face cognitive behavior therapy for social anxiety disorder and panic disorder: a randomized effectiveness trial. Behav Ther 2016 Mar;47(2):166-183. [CrossRef] [Medline]
    91. Nordgreen T, Standal B, Mannes H, Haug T, Sivertsen B, Carlbring P, et al. Guided self-help via internet for panic disorder: dissemination across countries. Comput Hum Behav 2010 Jul;26(4):592-596. [CrossRef]
    92. Knaevelsrud C, Maercker A. Internet-based treatment for PTSD reduces distress and facilitates the development of a strong therapeutic alliance: a randomized controlled clinical trial. BMC Psychiatry 2007 Apr 19;7:13 [FREE Full text] [CrossRef] [Medline]
    93. Lange A, Rietdijk D, Hudcovicova M, van de Ven JP, Schrieken B, Emmelkamp PM. Interapy: a controlled randomized trial of the standardized treatment of posttraumatic stress through the internet. J Consult Clin Psychol 2003 Oct;71(5):901-909. [CrossRef] [Medline]
    94. Ruwaard J, Schrieken B, Schrijver M, Broeksteeg J, Dekker J, Vermeulen H, et al. Standardized web-based cognitive behavioural therapy of mild to moderate depression: a randomized controlled trial with a long-term follow-up. Cogn Behav Ther 2009;38(4):206-221. [CrossRef] [Medline]
    95. Ruwaard J, Broeksteeg J, Schrieken B, Emmelkamp P, Lange A. Web-based therapist-assisted cognitive behavioral treatment of panic symptoms: a randomized controlled trial with a three-year follow-up. J Anxiety Disord 2010 May;24(4):387-396. [CrossRef] [Medline]
    96. Klein B, Richards JC, Austin DW. Efficacy of internet therapy for panic disorder. J Behav Ther Exp Psychiatry 2006 Sep;37(3):213-238. [CrossRef] [Medline]
    97. Richards JC, Klein B, Austin DW. Internet cognitive behavioural therapy for panic disorder: Does the inclusion of stress management information improve end-state functioning? Clin Psychol 2006;10(1):2-15. [CrossRef]
    98. Bergström J, Andersson G, Ljótsson B, Rück C, Andréewitch S, Karlsson A, et al. Internet-versus group-administered cognitive behaviour therapy for panic disorder in a psychiatric setting: a randomised trial. BMC Psychiatry 2010 Jul 2;10:54 [FREE Full text] [CrossRef] [Medline]
    99. Titov N, Fogliati VJ, Staples LG, Gandy M, Johnston K, Wootton B, et al. Treating anxiety and depression in older adults: randomised controlled trial comparing guided. Self-guided internet-delivered cognitive-behavioural therapy. BJPsych Open 2016 Jan;2(1):50-58 [FREE Full text] [CrossRef] [Medline]
    100. Wootton BM, Dear BF, Johnston L, Terides MD, Titov N. Remote treatment of obsessive-compulsive disorder: a randomized controlled trial. J Obsess-Compuls Rel 2013 Oct;2(4):375-384. [CrossRef]
    101. Spence J, Titov N, Johnston L, Jones MP, Dear BF, Solley K. Internet-based trauma-focused cognitive behavioural therapy for PTSD with and without exposure components: a randomised controlled trial. J Affect Disord 2014 Jun;162:73-80. [CrossRef] [Medline]
    102. Mattick RP, Clarke J. Development and validation of measures of social phobia scrutiny fear and social interaction anxiety. Behav Res Ther 1998 Apr;36(4):455-470. [CrossRef] [Medline]
    103. Baker SL, Heinrichs N, Kim H, Hofmann SG. The liebowitz social anxiety scale as a self-report instrument: a preliminary psychometric analysis. Behav Res Ther 2002 Jun;40(6):701-715. [CrossRef] [Medline]
    104. Kroenke K, Spitzer RL. The PHQ-9: a new depression diagnostic and severity measure. Psych Ann 2002 Sep 1;32(9):509-515. [CrossRef]
    105. Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001 Sep;16(9):606-613 [FREE Full text] [CrossRef] [Medline]
    106. Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006 May 22;166(10):1092-1097. [CrossRef] [Medline]
    107. Houck PR, Spiegel DA, Shear MK, Rucci P. Reliability of the self-report version of the panic disorder severity scale. Depress Anxiety 2002;15(4):183-185. [CrossRef] [Medline]
    108. Sheehan KH, Sheehan DV. Assessing treatment effects in clinical trials with the discan metric of the Sheehan disability scale. Int Clin Psychopharmacol 2008 Mar;23(2):70-83. [CrossRef] [Medline]
    109. Svanborg P, Asberg M. A new self-rating scale for depression and anxiety states based on the comprehensive psychopathological rating scale. Acta Psychiatr Scand 1994 Jan;89(1):21-28. [CrossRef] [Medline]
    110. Marks I, Mathews A. Brief standard self-rating for phobic patients. Behav Res Ther 1979;17(3):263-267. [CrossRef] [Medline]
    111. Beck AT, Ward CH, Mendelson M, Mock J, Erbaugh J. An inventory for measuring depression. Arch Gen Psychiatry 1961 Jun;4:561-571. [CrossRef] [Medline]
    112. Hamilton M. A rating scale for depression. J Neurol Neurosurg Psychiatry 1960 Feb;23:56-62 [FREE Full text] [CrossRef] [Medline]
    113. Beck AT, Epstein N, Brown G, Steer RA. An inventory for measuring clinical anxiety: psychometric properties. J Consult Clin Psychol 1988 Dec;56(6):893-897. [CrossRef] [Medline]
    114. Goodman WK, Price LH, Rasmussen SA, Mazure C, Fleischmann RL, Hill CL, et al. The Yale-Brown obsessive compulsive scale. I. Development, use, and reliability. Arch Gen Psychiatry 1989 Nov;46(11):1006-1011. [CrossRef] [Medline]
    115. Chambless DL, Caputo GC, Bright P, Gallagher R. Assessment of fear of fear in agoraphobics: the body sensations questionnaire and the agoraphobic cognitions questionnaire. J Consult Clin Psychol 1984 Dec;52(6):1090-1097. [CrossRef]
    116. Chambless DL, Caputo G, Jasin SE, Gracely EJ, Williams C. The mobility inventory for agoraphobia. Behav Res Ther 1985;23(1):35-44. [CrossRef]
    117. Nieuwenhuijsen K, de Boer AG, Verbeek JH, Blonk RW, van Dijk FJ. The depression anxiety stress Sscales (DASS): detecting anxiety disorder and depression in employees absent from work because of mental health problems. Occup Environ Med 2003 Jun;60(Suppl 1):i77-i82 [FREE Full text] [CrossRef] [Medline]
    118. Horowitz M, Wilner N, Alvarez W. Impact of event scale: a measure of subjective stress. Psychosom Med 1979 May;41(3):209-218. [CrossRef] [Medline]
    119. Taylor S, Cox BJ. Anxiety sensitivity: multiple dimensions and hierarchic structure. Behav Res Ther 1998 Jan;36(1):37-51. [CrossRef] [Medline]
    120. McDonald SD, Calhoun PS. The diagnostic accuracy of the PTSD checklist: a critical review. Clin Psychol Rev 2010 Dec;30(8):976-987. [CrossRef] [Medline]
    121. Borkovec TD, Nau SD. Credibility of analogue therapy rationales. J Behav Ther Exp Psychiatry 1972 Dec;3(4):257-260. [CrossRef]
    122. Devilly GJ, Borkovec TD. Psychometric properties of the credibility/expectancy questionnaire. J Behav Ther Exp Psychiatry 2000 Jun;31(2):73-86. [CrossRef] [Medline]
    123. Nguyen TD, Attkisson C, Stegner BL. Assessment of patient satisfaction: development and refinement of a service evaluation questionnaire. Eval Program Plann 1983;6(3-4):299-313. [CrossRef] [Medline]
    124. Karyotaki E, Ebert DD, Donkin L, Riper H, Twisk J, Burger S, et al. Do guided internet-based interventions result in clinically relevant changes for patients with depression? An individual participant data meta-analysis. Clin Psychol Rev 2018 Jul;63:80-92. [CrossRef] [Medline]
    125. Olthuis JV, Watt MC, Bailey K, Hayden JA, Stewart SH. Therapist-supported Internet cognitive behavioural therapy for anxiety disorders in adults. Cochrane Database Syst Rev 2015 Mar 05(3):CD011565. [CrossRef] [Medline]
    126. Seidler Z, Dawes A, Rice S, Oliffe J, Dhillon H. The role of masculinity in men's help-seeking for depression: a systematic review. Clin Psychol Rev 2016 Nov;49:106-118. [CrossRef] [Medline]
    127. Thompson A, Anisimowicz Y, Miedema B, Hogg W, Wodchis W, Aubrey-Bassler K. The influence of gender and other patient characteristics on health care-seeking behaviour: a QUALICOPC study. BMC Fam Pract 2016 Mar 31;17:38 [FREE Full text] [CrossRef] [Medline]
    128. Seedat S, Scott KM, Angermeyer MC, Berglund P, Bromet EJ, Brugha TS, et al. Cross-national associations between gender and mental disorders in the World Health Organization World Mental Health Surveys. Arch Gen Psychiatry 2009 Jul;66(7):785-795 [FREE Full text] [CrossRef] [Medline]
    129. Kuehner C. Gender differences in unipolar depression: an update of epidemiological findings and possible explanations. Acta Psychiatr Scand 2003 Sep;108(3):163-174. [CrossRef] [Medline]
    130. van Ballegooijen W, Cuijpers P, van Straten A, Karyotaki E, Andersson G, Smit J, et al. Adherence to Internet-based and face-to-face cognitive behavioural therapy for depression: a meta-analysis. PLoS One 2014;9(7):e100674 [FREE Full text] [CrossRef] [Medline]
    131. Richards D, Richardson T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev 2012 Jun;32(4):329-342. [CrossRef] [Medline]
    132. Ebert DD, Baumeister H. Internet- und mobil-basierte Interventionen in der Psychotherapiein Überblick. Psychotherapeutenjournal 2016;1:31. [CrossRef]
    133. Allen AR, Newby JM, Mackenzie A, Smith J, Boulton M, Loughnan SA, et al. Internet cognitive-behavioural treatment for panic disorder: randomised controlled trial and evidence of effectiveness in primary care. BJPsych Open 2016 Mar;2(2):154-162 [FREE Full text] [CrossRef] [Medline]
    134. Andersson G, Waara J, Jonsson U, Malmaeus F, Carlbring P, Ost LG. Internet-based exposure treatment versus one-session exposure treatment of snake phobia: a randomized controlled trial. Cogn Behav Ther 2013;42(4):284-291. [CrossRef] [Medline]
    135. Andersson G, Waara J, Jonsson U, Malmaeus F, Carlbring P, Ost LG. Internet-based self-help versus one-session exposure in the treatment of spider phobia: a randomized controlled trial. Cogn Behav Ther 2009;38(2):114-120. [CrossRef] [Medline]
    136. Berger T, Boettcher J, Caspar F. Internet-based guided self-help for several anxiety disorders: a randomized controlled trial comparing a tailored with a standardized disorder-specific approach. Psychotherapy (Chic) 2014 Jun;51(2):207-219. [CrossRef] [Medline]
    137. Berger T, Caspar F, Richardson R, Kneubühler B, Sutter D, Andersson G. Internet-based treatment of social phobia: a randomized controlled trial comparing unguided with two types of guided self-help. Behav Res Ther 2011 Mar;49(3):158-169. [CrossRef] [Medline]
    138. Berger T, Hohl E, Caspar F. Internet-based treatment for social phobia: a randomized controlled trial. J Clin Psychol 2009 Oct;65(10):1021-1035. [CrossRef] [Medline]
    139. Bergström J, Andersson G, Ljótsson B, Rück C, Andréewitch S, Karlsson A, et al. Internet-versus group-administered cognitive behaviour therapy for panic disorder in a psychiatric setting: a randomised trial. BMC Psychiatry 2010 Jul 2;10:54 [FREE Full text] [CrossRef] [Medline]
    140. Carlbring P, Bohman S, Brunt S, Buhrman M, Westling BE, Ekselius L, et al. Remote treatment of panic disorder: a randomized trial of internet-based cognitive behavior therapy supplemented with telephone calls. Am J Psychiatry 2006 Dec;163(12):2119-2125. [CrossRef] [Medline]
    141. Carlbring P, Gunnarsdóttir M, Hedensjö L, Andersson G, Ekselius L, Furmark T. Treatment of social phobia: randomised trial of internet-delivered cognitive-behavioural therapy with telephone support. Br J Psychiatry 2007 Feb;190:123-128. [CrossRef] [Medline]
    142. Carlbring P, Maurin L, Törngren C, Linna E, Eriksson T, Sparthan E, et al. Individually-tailored, internet-based treatment for anxiety disorders: a randomized controlled trial. Behav Res Ther 2011 Jan;49(1):18-24. [CrossRef] [Medline]
    143. Furmark T, Carlbring P, Hedman E, Sonnenstein A, Clevberger P, Bohman B, et al. Guided and unguided self-help for social anxiety disorder: randomised controlled trial. Br J Psychiatry 2009 Nov;195(5):440-447. [CrossRef] [Medline]
    144. Hedman E, Andersson G, Ljótsson B, Andersson E, Rück C, Mörtberg E, et al. Internet-based cognitive behavior therapy vs. cognitive behavioral group therapy for social anxiety disorder: a randomized controlled non-inferiority trial. PLoS One 2011 Mar 25;6(3):e18001 [FREE Full text] [CrossRef] [Medline]
    145. Hedman E, El Alaoui S, Lindefors N, Andersson E, Rück C, Ghaderi A, et al. Clinical effectiveness and cost-effectiveness of Internet- vs. group-based cognitive behavior therapy for social anxiety disorder: 4-year follow-up of a randomized trial. Behav Res Ther 2014 Aug;59:20-29. [CrossRef] [Medline]
    146. Ivarsson D, Blom M, Hesser H, Carlbring P, Enderby P, Nordberg R, et al. Guided internet-delivered cognitive behavior therapy for post-traumatic stress disorder: a randomized controlled trial. Internet Interv 2014 Mar;1(1):33-40. [CrossRef]
    147. Johnston L, Titov N, Andrews G, Spence J, Dear BF. A RCT of a transdiagnostic internet-delivered treatment for three anxiety disorders: examination of support roles and disorder-specific outcomes. PLoS One 2011;6(11):e28079 [FREE Full text] [CrossRef] [Medline]
    148. Jones SL, Hadjistavropoulos HD, Soucy JN. A randomized controlled trial of guided internet-delivered cognitive behaviour therapy for older adults with generalized anxiety. J Anxiety Disord 2016 Jan;37:1-9. [CrossRef] [Medline]
    149. Kok RN, van Straten A, Beekman AT, Cuijpers P. Short-term effectiveness of web-based guided self-help for phobic outpatients: randomized controlled trial. J Med Internet Res 2014 Sep 29;16(9):e226 [FREE Full text] [CrossRef] [Medline]
    150. Nordgren LB, Hedman E, Etienne J, Bodin J, Kadowaki A, Eriksson S, et al. Effectiveness and cost-effectiveness of individually tailored internet-delivered cognitive behavior therapy for anxiety disorders in a primary care population: a randomized controlled trial. Behav Res Ther 2014 Aug;59:1-11 [FREE Full text] [CrossRef] [Medline]
    151. Oromendia P, Orrego J, Bonillo A, Molinuevo B. Internet-based self-help treatment for panic disorder: a randomized controlled trial comparing mandatory versus optional complementary psychological support. Cogn Behav Ther 2016 Jun;45(4):270-286. [CrossRef] [Medline]
    152. Paxling B, Almlöv J, Dahlin M, Carlbring P, Breitholtz E, Eriksson T, et al. Guided internet-delivered cognitive behavior therapy for generalized anxiety disorder: a randomized controlled trial. Cogn Behav Ther 2011;40(3):159-173. [CrossRef] [Medline]
    153. Robinson E, Titov N, Andrews G, McIntyre K, Schwencke G, Solley K. Internet treatment for generalized anxiety disorder: a randomized controlled trial comparing clinician vs technician assistance. PLoS One 2010 Jun 3;5(6):e10942 [FREE Full text] [CrossRef] [Medline]
    154. Silfvernagel L, Carlbring P, Kabo J, Edström S, Eriksson J, Månson L, et al. Individually tailored internet-based treatment for young adults and adults with panic attacks: randomized controlled trial. J Med Internet Res 2012 Jun 26;14(3):e65 [FREE Full text] [CrossRef] [Medline]
    155. Spence J, Titov N, Dear BF, Johnston L, Solley K, Lorian C, et al. Randomized controlled trial of Internet-delivered cognitive behavioral therapy for posttraumatic stress disorder. Depress Anxiety 2011 Jul;28(7):541-550. [CrossRef] [Medline]
    156. Titov N, Andrews G, Choi I, Schwencke G, Mahoney A. Shyness 3: randomized controlled trial of guided versus unguided internet-based CBT for social phobia. Aust N Z J Psychiatry 2008 Dec;42(12):1030-1040. [CrossRef] [Medline]
    157. Titov N, Andrews G, Johnston L, Robinson E, Spence J. Transdiagnostic internet treatment for anxiety disorders: a randomized controlled trial. Behav Res Ther 2010 Sep;48(9):890-899. [CrossRef] [Medline]
    158. Titov N, Dear BF, Schwencke G, Andrews G, Johnston L, Craske MG, et al. Transdiagnostic internet treatment for anxiety and depression: a randomised controlled trial. Behav Res Ther 2011 Aug;49(8):441-452. [CrossRef] [Medline]
    159. Tulbure BT, Szentagotai A, David O, Ștefan S, Månsson KN, David D, et al. Internet-delivered cognitive-behavioral therapy for social anxiety disorder in Romania: a randomized controlled trial. PLoS One 2015;10(5):e0123997 [FREE Full text] [CrossRef] [Medline]
    160. van Ballegooijen W, Riper H, Klein B, Ebert DD, Kramer J, Meulenbeek P, et al. An internet-based guided self-help intervention for panic symptoms: randomized controlled trial. J Med Internet Res 2013 Jul 29;15(7):e154 [FREE Full text] [CrossRef] [Medline]
    161. Wims E, Titov N, Andrews G, Choi I. Clinician-assisted Internet-based treatment is effective for panic: a randomized controlled trial. Aust N Z J Psychiatry 2010 Jul;44(7):599-607. [CrossRef] [Medline]
    162. Wootton BM, Dear BF, Johnston L, Terides MD, Titov N. Remote treatment of obsessive-compulsive disorder: a randomized controlled trial. J Obsess-Compuls Rel 2013 Oct;2(4):375-384. [CrossRef]
    163. Stewart RE, Chambless DL. Cognitive-behavioral therapy for adult anxiety disorders in clinical practice: a meta-analysis of effectiveness studies. J Consult Clin Psychol 2009 Aug;77(4):595-606. [CrossRef] [Medline]
    164. Ebert DD, Donkin L, Andersson G, Andrews G, Berger T, Carlbring P, et al. Does internet-based guided-self-help for depression cause harm? An individual participant data meta-analysis on deterioration rates and its moderators in randomized controlled trials. Psychol Med 2016 Oct;46(13):2679-2693 [FREE Full text] [CrossRef] [Medline]
    165. Rozental A, Magnusson K, Boettcher J, Andersson G, Carlbring P. For better or worse: an individual patient data meta-analysis of deterioration among participants receiving internet-based cognitive behavior therapy. J Consult Clin Psychol 2017 Feb;85(2):160-177. [CrossRef] [Medline]
    166. Cuijpers P, Reijnders M, Karyotaki E, de Wit L, Ebert DD. Negative effects of psychotherapies for adult depression: a meta-analysis of deterioration rates. J Affect Disord 2018 Oct 15;239:138-145. [CrossRef] [Medline]
    167. Rozental A, Castonguay L, Dimidjian S, Lambert M, Shafran R, Andersson G, et al. Negative effects in psychotherapy: commentary and recommendations for future research and clinical practice. BJPsych Open 2018 Jul;4(4):307-312 [FREE Full text] [CrossRef] [Medline]
    168. Mewton L, Andrews G. Cognitive behaviour therapy via the internet for depression: a useful strategy to reduce suicidal ideation. J Affect Disord 2015 Jan 1;170:78-84. [CrossRef] [Medline]
    169. Kendrick T, El-Gohary M, Stuart B, Gilbody S, Churchill R, Aiken L, et al. Routine use of patient reported outcome measures (PROMs) for improving treatment of common mental health disorders in adults. Cochrane Database Syst Rev 2016 Jul 13;7:CD011119 [FREE Full text] [CrossRef] [Medline]
    170. Lambert MJ, Harmon KL. The merits of implementing routine outcome monitoring in clinical practice. Clin Psychol Sci Pract 2018 Dec 12;25(4):e12268. [CrossRef]
    171. Boswell JF, Kraus DR, Miller SD, Lambert MJ. Implementing routine outcome monitoring in clinical practice: benefits, challenges, and solutions. Psychother Res 2015;25(1):6-19. [CrossRef] [Medline]
    172. Cuijpers P, Andersson G, Donker T, van Straten A. Psychological treatment of depression: results of a series of meta-analyses. Nord J Psychiatry 2011 Dec;65(6):354-364. [CrossRef] [Medline]
    173. Mason E, Andrews G. The use of automated assessments in internet-based CBT: the computer will be with you shortly. Internet Interv 2014 Oct;1(4):216-224. [CrossRef]
    174. Cuijpers P. The challenges of improving treatments for depression. J Am Med Assoc 2018 Dec 25;320(24):2529-2530. [CrossRef] [Medline]
    175. Titov N, Dear B, Nielssen O, Staples L, Hadjistavropoulos H, Nugent M, et al. ICBT in routine care: a descriptive analysis of successful clinics in five countries. Internet Interv 2018 Sep;13:108-115 [FREE Full text] [CrossRef] [Medline]
    176. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011 Mar;38(2):65-76 [FREE Full text] [CrossRef] [Medline]
    177. Hermes ED, Lyon AR, Schueller SM, Glass JE. Measuring the implementation of behavioral intervention technologies: recharacterization of established outcomes. J Med Internet Res 2019 Jan 25;21(1):e11752 [FREE Full text] [CrossRef] [Medline]
    178. Klein JP, Knaevelsrud C, Bohus M, Ebert DD, Gerlinger G, Günther K, et al. [Finding the needles in the haystack: quality criteria for the use of internet-based self-management interventions in the prevention and treatment of mental disorders]. Fortschr Neurol Psychiatr 2019 Mar;87(3):187-191. [CrossRef] [Medline]


    Abbreviations

    GP: general practitioner
    iCBT: internet-based cognitive behavioral therapy
    RCT: randomized controlled trial


    Edited by G Eysenbach; submitted 03.02.20; peer-reviewed by A Rozental, E Szigethy, K Magnusson, N Lindefors; comments to author 26.02.20; revised version received 06.05.20; accepted 03.06.20; published 31.08.20

    ©Anne Etzelmueller, Christiaan Vis, Eirini Karyotaki, Harald Baumeister, Nickolai Titov, Matthias Berking, Pim Cuijpers, Heleen Riper, David Daniel Ebert. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 31.08.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.