This paper is in the following e-collection/theme issue:

Cost-Effectiveness and Economics (234) Registered Report (528) Depression and Mood Disorders; Suicide Prevention (1290)

Published on in Vol 21, No 10 (2019): October

Cost and Effectiveness of Blended Versus Standard Cognitive Behavioral Therapy for Outpatients With Depression in Routine Specialized Mental Health Care: Pilot Randomized Controlled Trial

Cost and Effectiveness of Blended Versus Standard Cognitive Behavioral Therapy for Outpatients With Depression in Routine Specialized Mental Health Care: Pilot Randomized Controlled Trial

Cost and Effectiveness of Blended Versus Standard Cognitive Behavioral Therapy for Outpatients With Depression in Routine Specialized Mental Health Care: Pilot Randomized Controlled Trial

Authors of this article:

Lisa Catharine Kooistra1, 2, 3 Author Orcid Image ;   Jenneke Elize Wiersma2, 3 Author Orcid Image ;   Jeroen Ruwaard2, 3 Author Orcid Image ;   Koen Neijenhuijs1, 3, 4 Author Orcid Image ;   Joran Lokkerbol5 Author Orcid Image ;   Patricia van Oppen2, 3, 6 Author Orcid Image ;   Filip Smit1, 3, 5, 7 Author Orcid Image ;   Heleen Riper1, 2, 3, 6 Author Orcid Image
Article Authors Cited by (50) Tweetations (16) Metrics

    Original Paper

    1Department of Clinical, Neuro and Developmental Psychology, Clinical Psychology Section, Vrije Universiteit Amsterdam, Amsterdam, Netherlands

    2Department of Research and Innovation, GGZ in Geest/Amsterdam University Medical Center, VU University Medical Center, Amsterdam, Netherlands

    3Amsterdam Public Health Research Institute, Amsterdam University Medical Center, VU University Medical Center, Amsterdam, Netherlands

    4Cancer Center Amsterdam, Amsterdam University Medical Center, VU University Medical Center, Amsterdam, Netherlands

    5Center of Economic Evaluation, Trimbos Institute (Netherlands Institute of Mental Health and Addiction), Utrecht, Netherlands

    6Department of Psychiatry, GGZ in Geest/Amsterdam University Medical Center, VU University Medical Center, Amsterdam, Netherlands

    7Department of Epidemiology and Biostatistics, Amsterdam University Medical Center, VU University Medical Center, Amsterdam, Netherlands

    Corresponding Author:

    Lisa Catharine Kooistra, MSc

    Department of Research and Innovation

    GGZ in Geest/Amsterdam University Medical Center, VU University Medical Center

    Postbus 74077

    Amsterdam, 1070 BB

    Netherlands

    Phone: 31 204448180

    Email: l.kooistra@ggzingeest.nl


    Background: Cognitive behavioral therapy (CBT) is an effective treatment, but access is often restricted due to costs and limited availability of trained therapists. Blending online and face-to-face CBT for depression might improve cost-effectiveness and treatment availability.

    Objective: This pilot study aimed to examine the costs and effectiveness of blended CBT compared with standard CBT for depressed patients in specialized mental health care to guide further research and development of blended CBT.

    Methods: Patients were randomly allocated to blended CBT (n=53) or standard CBT (n=49). Blended CBT consisted of 10 weekly face-to-face sessions and 9 Web-based sessions. Standard CBT consisted of 15 to 20 weekly face-to-face sessions. At baseline and 10, 20, and 30 weeks after start of treatment, self-assessed depression severity, quality-adjusted life-years (QALYs), and costs were measured. Clinicians, blinded to treatment allocation, assessed psychopathology at all time points. Data were analyzed using linear mixed models. Uncertainty intervals around cost and effect estimates were estimated with 5000 Monte Carlo simulations.

    Results: Blended CBT treatment duration was mean 19.0 (SD 12.6) weeks versus mean 33.2 (SD 23.0) weeks in standard CBT (P<.001). No significant differences were found between groups for depressive episodes (risk difference [RD] 0.06, 95% CI −0.05 to 0.19), response to treatment (RD 0.03, 95% CI −0.10 to 0.15), and QALYs (mean difference 0.01, 95% CI −0.03 to 0.04). Mean societal costs for blended CBT were €1183 higher than standard CBT. This difference was not significant (95% CI −399 to 2765). Blended CBT had a probability of being cost-effective compared with standard CBT of 0.02 per extra QALY and 0.37 for an additional treatment response, at a ceiling ratio of €25,000. For health care providers, mean costs for blended CBT were €176 lower than standard CBT. This difference was not significant (95% CI −659 to 343). At €0 per additional unit of effect, the probability of blended CBT being cost-effective compared with standard CBT was 0.75. The probability increased to 0.88 at a ceiling ratio of €5000 for an added treatment response, and to 0.85 at €10,000 per QALY gained. For avoiding new depressive episodes, blended CBT was deemed not cost-effective compared with standard CBT because the increase in costs was associated with negative effects.

    Conclusions: This pilot study shows that blended CBT might be a promising way to engage depressed patients in specialized mental health care. Compared with standard CBT, blended CBT was not considered cost-effective from a societal perspective but had an acceptable probability of being cost-effective from the health care provider perspective. Results should be carefully interpreted due to the small sample size. Further research in larger replication studies focused on optimizing the clinical effects of blended CBT and its budget impact is warranted.

    Trial Registration: Netherlands Trial Register NTR4650; https://www.trialregister.nl/trial/4408

    International Registered Report Identifier (IRRID): RR2-10.1186/s12888-014-0290-z

    J Med Internet Res 2019;21(10):e14261

    doi:10.2196/14261

    Keywords

    depression; blended cognitive behavioral therapy; specialized mental health care; cost-effectiveness; randomized controlled trial 


    Several evidence-based pharmacological and psychological treatments have been developed for major depressive disorder (MDD) [1,2]. Within the domain of psychotherapy, there is an especially large body of evidence supporting the efficacy of cognitive behavioral therapy (CBT) [3]. Unfortunately, there is a discrepancy between treatment availability and treatment demand [4]. This can partly be explained by increasingly insufficient mental health care budgets, which limit the availability of trained psychotherapists who can provide evidence-based treatments [5]. As a consequence, patients with severe symptoms often do not receive treatment, or they have to be placed on waiting lists rather than having immediate access to specialized depression care [4,6]. Therefore, there is a high need for efficient and cost-effective mental health care to manage this problem [7].

    Web-based (online) psychotherapy is often cited as a promising way to reduce treatment costs and increase treatment availability for common mental disorders [8-11]. For example, it may lower the required therapist time per patient [12]. The efficacy of several online treatments has been demonstrated for the treatment of depression when compared with control groups [13-15]. Evidence suggests that therapist-guided online treatment can be equally as effective as standard face-to-face therapy [16,17]. The cost-effectiveness of guided online treatment has been less well studied. An individual patient data meta-analysis, combining data from five randomized controlled trials (RCTs), suggested that guided online treatment is not yet cost-effective compared with control conditions [18].

    To date, most studies have focused on community samples and self-referred participants. Less is known about the costs and effects of online therapy in routine specialized mental health care for patients with more severe or complex depression profiles. This patient group is likely to require more personalized treatment, such as monitoring of suicidal ideation or addressing comorbid disorders [16]. The integration of online and face-to-face treatment into a blended treatment format could be a promising way to address these matters [19-23]. Blended treatment allows therapists to closely monitor their patients, both in face-to-face sessions at the clinic and in an online environment. At the same time, it is thought to retain the positive aspects associated with online treatment, such as lower costs compared with standard treatment, reduction of therapist time, and increased patient self-management [19,24,25].

    Although several studies are in progress, current evidence for blended treatment is limited [26-29]. Most studies to date have evaluated the online component as an add-on to standard care rather than offering an integrated blended treatment protocol [30-35]. Results of these studies are promising and indicate that blended treatment may be a viable treatment option. However, the treatment format limits the margin for cost-effectiveness because therapist time is not reduced and overall treatment dosage may even increase [18,36]. Recently, Thase and colleagues [25] used a more integrated approach to blended treatment. Therapist time was limited by combining nine online sessions with twelve 25-minute (instead of 50-minute) face-to-face sessions. Compared with CBT (n=77), blended CBT (n=77) led to noninferior results on the Hamilton Depression Rating Scale (HAM-D) in medication-free adults with depression, when provided in two university clinics.

    This study compares the costs and effects of integrated blended CBT with standard CBT when provided to depressed patients in the acute phase of treatment in specialized mental health care. The blended treatment aimed to replace half of face-to-face treatments with online sessions, and thereby shorten treatment duration when compared with standard CBT for MDD [22]. The study examines whether blended CBT has the potential to lead to comparable clinical effects as standard CBT at lower costs. The study was designed as a pilot study with the aim to guide further development of blended CBT and inform future research on feasibility, effectiveness, and cost-effectiveness of blended CBT for depression in outpatient specialized mental health care.


    Study Design and Participants

    This pilot study was designed as an RCT in three mental health care organizations in the Netherlands, at five outpatient treatment locations. Blended CBT was compared with standard CBT for depression. Outcomes were measured before treatment allocation at baseline and 10, 20, and 30 weeks after start of treatment. Adult patients with a Diagnostic and Statistical Manual of Mental Disorders (Fourth Edition, Text Revision; DSM-IV-TR) diagnosis [37] of MDD, who were indicated for individual CBT by the local intake staff, were recruited during the intake procedure. In the Netherlands, patients can be referred to specialized services when they do not respond to treatment in primary mental health care [38]. Therefore, patients in this trial were likely to have complex or severe clinical profiles and to have undergone some form of pharmacotherapy or short-term psychotherapy before this study.

    Exclusion criteria were inadequate proficiency in the Dutch language; no valid email address or no computer with internet access; current psychotic disorder, bipolar disorder, or substance dependence; or high risk for suicide (current plans). Patients with current substance dependence could participate in the trial if they reported abstinence and had been treated at a center specialized in treatment of substance abuse for at least a month before random allocation.

    Diagnoses and suicidal ideation were assessed with the Mini-International Neuropsychiatric Interview Plus (MINI-Plus) [39,40]. Assessors were trained research assistants with master’s degrees in psychology. The study protocol has been published [41]. The trial was registered in the Netherlands Trial Register (registration number NTR4650) and approved by the Medical Ethics Committee of the VU University Medical Center Amsterdam (registration number 2014.191).

    Randomization and Masking

    Patients were randomly allocated to blended CBT or standard CBT when they met the inclusion criteria, provided a signed informed consent form, and completed the full baseline assessment. Allocation was stratified by mental health care center using a computer-generated random number table. An independent researcher conducted the randomization. Patients and therapists were aware of group allocation, but allocation was concealed from outcome assessors.

    Interventions

    Both interventions consisted of cognitive behavioral depression treatment. The content of CBT was based on the standard treatment manual [42-44], which recommends 15 to 20 weekly sessions focusing on psychoeducation, behavioral activation, cognitive restructuring, and relapse prevention. As part of the treatment, patients in both treatment groups were encouraged to fill in the 16-item Quick Inventory of Depressive Symptomatology Self-Report on a weekly basis to allow patients and their therapists to monitor change in depression severity [45]. Patients who completed 14 sessions or more (75% of 18 sessions) were considered treatment completers. Parallel pharmacological treatment was allowed.

    Blended CBT consisted of 10 face-to-face sessions at the specialized mental health care center and 9 Web-based (online) sessions that patients worked through at home. Therapists and patients had personal, password-protected accounts on the Web-based treatment platform. Therapists were trained in the use of the Web-based treatment platform and received a treatment manual. Patients received information on how to work with the platform in the first face-to-face session. After each online session, therapists provided online therapeutic feedback regarding content and progress. Therapists could let patients repeat an online session if this was warranted. The blended CBT protocol aimed to provide one face-to-face session, one online session, and one online feedback message per week over 10 weeks, starting with a face-to-face session. The intervention was semistructured, offering online sessions in a fixed order and providing therapists with a protocol for each face-to-face session. Therapists were allowed to personalize face-to-face sessions to each patient’s individual needs and situation in terms of themes and techniques. More detailed information on the format and content of blended CBT can be found elsewhere [22,41].

    The standard CBT group received individual face-to-face cognitive behavioral depression treatment at the clinic in accordance with routine care procedures. The content of standard CBT was comparable to the content of blended CBT. Therapists were advised to plan weekly sessions but were allowed to deviate from the treatment manual when necessary. Based on the guideline, standard CBT was expected to consist of 18 sessions on average, provided over approximately 20 weeks. However, duration of treatment could vary per patient.

    Outcomes

    The primary clinical outcome was self-reported depression severity at 10, 20, and 30 weeks after the start of treatment, as measured by a Web-based version of the Inventory of Depressive Symptomatology Self-Report (IDS-SR30) [46-49]. Within the cost-effectiveness framework, the IDS-SR30 scores were used to assess treatment response based on the reliable change index (RCI) [50]. Baseline IDS-SR30 scores were subtracted from follow-up scores and then divided by the standard error of the difference scores (SE 4.78, Cronbach alpha=.84). Treatment response (reliable change) was coded as absent (RCI≥−1.96) versus present (RCI<−1.96). Remission was defined as the combination of an RCI less than −1.96 and an IDS-SR30 score less than 13, indicating no depression severity on the IDS-SR30 severity index [46,51].

    Clinician-rated outcomes included the presence of a depressive episode at 10, 20, and 30 weeks after the start of treatment and during follow-up. Depression diagnosis was assessed at 10, 20, and 30 weeks after the start of treatment during a telephone interview with section A of the MINI-Plus diagnostic interview [39,40] by trained research assistants who were blinded to treatment allocation. At baseline and 30-week follow-up, all sections of the MINI-Plus interview were administered during a face-to-face or telephone interview to assess comorbid diagnoses.

    To calculate quality-adjusted life-years (QALYs), the EQ-5D three-level version [52] questionnaire was administered online at all four time points. QALYs gained over the 30-week study period were estimated by linearly interpolating EQ-5D utility over the time points and correcting for time. Utility scores were based on the Dutch tariff [53].

    Self-reported resource use in the 4 weeks before each assessment was measured with a Web-based version of the Trimbos/iMTA questionnaire for Costs associated with Psychiatric Illness (TiC-P) [54]. The TiC-P includes questions on (1) the number of visits to various health care providers in primary and specialized care settings; (2) medication use for sleep, depression, and anxiety; (3) help from friends and family; and (4) absenteeism and reduced work productivity (presenteeism) in paid and unpaid work. For the main economic evaluation, costs were estimated from a societal perspective, which included all costs measured by the TiC-P. The health care provider perspective only included direct medical costs. A full overview of all cost categories can be found in Multimedia Appendix 1. Costs were computed by multiplying the number of units (contacts, visits, sessions) by the standard unit cost prices (for the year 2014) as reported in the most recent Dutch guideline for economic evaluations [55]. When costs were not included in the latest version of the manual, the cost price was derived from the previous guideline [56] and indexed to the year 2014. Medication prices were retrieved from the National Healthcare Institute (Zorginstituut Nederland) (Z-index [57]). For each prescription, the standard dispensing fee for pharmacists of €6 was added. For blended CBT patients, costs associated with the actual number of online feedback messages received were added, based on an estimated 30 minutes of therapist time per feedback message. Cumulative costs over the 30-week study period were estimated using linear interpolation.

    Statistical Analyses

    For descriptive purposes, clinical and cost outcomes were first examined in separate linear mixed-effect models. To account for missing data and the correlation between follow-up time points, linear mixed-effect models with restricted maximum likelihood were used to estimate the treatment effects, EQ-5D utility gained, and cumulative costs across time. The models included both fixed and random effects, which allowed for estimation on a patient level of how far each patient diverged from the fixed (group level) effect over time [58,59]. Main outcome measures (ie, IDS-SR depression severity, reliable change, MINI depression status, EQ-5D QALYs, and costs) were evaluated in separate mixed models. Time was included as a categorical variable (0, 10, 20, and 30 weeks). A logistic mixed-effects model was fitted for diagnosis of depressive episodes (present versus absent) and reliable improvement of depression severity (improvement versus no improvement). In these models, the random intercept was dropped from the model because all patients started at the same baseline value (a current diagnosis of depression).

    For the cost-effectiveness analyses, baseline adjusted linear mixed-effect models were estimated with group as the independent variable and societal costs, direct medical costs, and reliable change in depression severity as the dependent variables and a random effect across individuals. For dichotomous outcomes, linear mixed-effect models were fitted to estimate risk difference and calculate the incremental cost-effectiveness ratio (ICER). EQ-5D QALYs gained during the study were used as the dependent variable for the cost-utility analysis. Within the cost-effectiveness and cost-utility framework, time was not included in the regression models, rendering one estimate for cost, effectiveness, or utility for all follow-ups. Societal costs during the study period were used as the main cost outcome. Direct health care costs were examined as a separate cost outcome to conduct the health-economic evaluation from the health care system perspective. Uncertainty intervals of 95% around linear mixed-effect model regression estimates for group were estimated with 5000 probabilistic Monte Carlo simulations, running all mixed models simultaneously.

    The resulting pairs of cost and effect or utility estimates were plotted on cost-effectiveness and cost-utility planes. The y-axis represented relative costs, and the x-axis represented relative effects associated with blended CBT versus standard CBT. The axes divide the plane into four quadrants. In the northeast quadrant, blended CBT was more expensive and more effective than standard CBT. In the northwest quadrant, blended CBT was more expensive and less effective than status quo (standard CBT), meaning that blended CBT was dominated by standard CBT. In the southwest quadrant, blended CBT was less expensive and less effective than standard CBT. In the southeast quadrant, blended CBT was less expensive and more effective than standard CBT, which indicated that blended CBT dominated standard CBT. Cost-effectiveness acceptability curves were estimated to assess the probability of blended CBT being cost-effective given various willingness-to-pay ceilings.

    The mixed models were estimated using the lmer and glmer functions from the lme4 package (version 1.1-15) [60] using R software (version 3.4.4) [61]. Combined cost-effectiveness and cost-utility analyses were performed in Stata version 14.2 (StataCorp LP, College Station, TX, USA).

    Two sensitivity analyses were conducted. First, a linear mixed-effect model was estimated for societal costs excluding the costs of in-patient care (both in general hospitals and mental health care). Although these events are rare, they can act as influential outliers due to the high costs associated with in-patient care. Second, direct nonmedical costs associated with absenteeism and presenteeism were evaluated as a separate cost outcome to assess their contribution to the overall societal costs as evaluated in the main analysis.

    Power

    Sample size estimation was based on the probability of blended CBT being cost-effective in comparison with standard CBT for various willingness-to-pay ceilings (ie, the maximum additional financial contribution society is willing to invest to gain one more unit of treatment effect) [62]. Through a simulation study, the impact of different sample sizes on the stability of the cost-effectiveness acceptability curve was determined [62]. Using realistic fixed estimates of the mean and standard deviation of effects and the costs on the population level (based on Hakkaart-van Roijen et al [63]), a large number of trials were simulated in which sample sizes were systematically varied between n=10 to n=500 per group. At a sample size of n=75 per group, the probability estimates converged to acceptable 75% values within the relevant range of willingness-to-pay ceilings. Therefore, this study aimed for a sample size of N=150.


    Overview

    Between August 2014 and May 2016, 103 patients were randomized to blended CBT (n=54) or standard CBT (n=49). One patient withdrew from the study before commencing treatment due to starting another treatment elsewhere. The trial profile is presented in Figure 1. The full intention-to-treat sample consisted of 102 patients (blended CBT: n=53; standard CBT: n=49). This sample size was smaller than the initial goal of 150 patients [41]. The recruitment and screening of sufficient patients to randomize 150 patients proved to be unfeasible within the fixed time frame of this study. Fewer patients were referred to specialized services than expected. This was partly due to a nationwide reorganization of Dutch mental health care, which meant that a larger proportion of depressed patients were treated in primary mental health care rather than specialized care.

    ‎
    Figure 1. Flowchart. bCBT: blended cognitive behavioral therapy; CBT: standard cognitive behavioral therapy.
    View this figure

    Information on patients’ baseline demographic characteristics, clinical profiles, and costs are presented in Table 1. A notable number of patients (69 of 102, 68%) were diagnosed with at least one other DSM-IV disorder in addition to MDD, 60% (61 of 102) reported suicidal ideation, and 28% (29 of 102) had attempted suicide at some point in their lives. Patients in blended CBT showed a higher baseline depression severity (45.2 versus 41.5), lower mean utility score (0.36 versus 0.46), and higher direct medical costs (€989 versus €439) in the 4 weeks before baseline than patients in standard CBT.

    Table 1. Sample characteristics of participants at baseline (N=102).
    Patient characteristicsBlended CBTa (n=53)Standard CBT (n=49)Total (N=102)
    Demographic



    		Gender (female), n (%)35 (66)29 (60)64 (63)

    		Age (years), mean (SD)39.3 (11.3)38.1 (10.6)38.8 (10.9)

    		In a relationship, n (%)30 (57)30 (61)60 (59)

    		Education, n (%)




    		Low6 (11)2 (4)8 (8)


    		Middle30 (57)33 (67)63 (62)


    		High17 (32)14 (29)31 (30)

    		Employed, n (%)31 (59)28 (57)59 (58)

    		Nationality (Dutch), n (%)48 (91)47 (96)95 (93)
    Clinical characteristics



    		Blended CBT treatment preference, n (%)33 (62)33 (67)66 (65)

    		Any comorbidity,b n (%)38 (72)31 (63)69 (68)

    		Anxiety disorders, n (%)30 (57)25 (51)55 (54)

    		Other disorders,c n (%)25 (47)23 (47)48 (47)

    		Comorbid disorders,d mean (SD)1.8 (1.7)1.8 (1.9)1.8 (1.7)

    		MDDe duration in months, mean (SD)73.7 (73.2)80.2 (69.8)76.9 (70.9)

    		Depression (IDS-SR)f, mean (SD)45.2 (12.1)41.5 (11.6)43.4 (11.9)

    		Severity (IDS-SR), n (%)




    		Mild/moderate17 (32)22 (45)39 (38)


    		Severe/very severe36 (68)27 (55)64 (63)

    		Utility scores, mean (SD)0.36 (0.29)0.46 (0.31)0.41 (0.30)
    Prior treatment (4 weeks), n (%)



    		Antidepressant medication36 (68)29 (59)65 (64)

    		Psychological treatment44 (83)42 (86)86 (84)
    Costs (€, TiC-Pg, 4 weeks), mean (SD)



    		Direct medical989 (1626)439 (398)725 (1230)

    		Direct nonmedical428 (802)246 (526)339 (685)

    		Indirect nonmedical796 (1629)716 (1610)757 (1613)

    		Societal costs2205 (2615)1401 (1762)1819 (2271)

    aCBT: cognitive behavioral therapy.

    bThe DSM-IV-TR and ICD-10 codes are available on request.

    cObsessive compulsive disorder, posttraumatic stress disorder, alcohol/drug abuse, somatoform disorders.

    dAnxiety disorders: social phobia, panic with or without agoraphobia, agoraphobia, generalized anxiety disorder.

    eMDD: major depressive disorder.

    fIDS-SR: Inventory of Depressive Symptomatology, self-report version.

    gTiC-P: Trimbos/iMTA questionnaire for Costs associated with Psychiatric Illness.

    Study Dropout

    Overall, the MINI-Plus diagnostic interview could be administered to 85 of 102 patients (83%) at 10 weeks, 84 patients (82%) at 20 weeks, and 76 patients (75%) at 30 weeks. The online self-report questionnaires were completed by 77 patients at 10 weeks (76%), 65 patients (64%) at 20 weeks, and 65 patients (64%) at 30 weeks.

    Patients with missing data at one or more follow-up assessments (n=49) were on average five years younger than patients who completed all assessments (n=54; mean 36.0, SD 10.5 years versus mean 41.2, SD 10.8 years; t100=2.49, P=.02). Within the blended CBT group, missing data were not associated with patient characteristics. Patients with missing data in the standard CBT group were more often unemployed than patients who did not have missing data (χ22=14.1, P<.001). Figure 1 displays study and treatment adherence for both groups throughout the study period.

    Treatment Adherence

    Of 102 patients, 97 (95%) started treatment and received at least one session. In blended CBT, the average number of sessions matched the planned number of 19.5 sessions (SD 8.3, range 0 to 31). On average, blended CBT had a mean of 9.6 online sessions (SD 4.4, range 0 to 16) and 10.0 face-to-face sessions (SD 4.6, range 0 to 16). Per-patient online feedback was provided a mean of 8.4 times (SD 4.2, range 0 to 15). Based on the cutoff of 14 sessions (75% of the optimal blended CBT protocol of 19 sessions), 43 of 53 blended CBT patients (81%) were considered treatment completers. When the 75% completion criterion was applied to online and face-to-face sessions separately (>7 each), 40 blended CBT patients (75%) were considered completers. Seven patients (13%) completed the full blended CBT protocol within 10 weeks.

    In standard CBT, the mean number of sessions was 13.3 (SD 6.3, range 0 to 27), which was less than planned (15-18 sessions) within the time frame of this study. Based on the cutoff of 14 sessions, 22 of 49 patients (45%) were considered treatment completers. Four patients (8%) completed the standard CBT protocol (16-20 sessions) within 20 weeks.

    Compared with patients in the standard CBT group, blended CBT patients received significantly more sessions in total (19 versus 13, t100=−4.09, P<.001), but significantly fewer face-to-face sessions at the clinic (10 versus 13, t100=3.07, P=.003). As expected based on the treatment protocol, mean treatment duration was significantly shorter in blended CBT than in standard CBT, with an average of 19.0 (SD 12.6) weeks versus 33.2 (SD 23.0) weeks (t100=3.91, P<.001). When therapist time spent on online feedback was included, the combined amount of direct and indirect therapist time did not differ between groups with an average of 14.0 (SD 6.2) hours for blended CBT and 13.3 (SD 6.3) hours for standard CBT (t100=−0.55, P=.58).

    Clinical Outcomes

    Deviation between the desired time points and actual time of data collection was deemed within an acceptable range to enter time as a fixed categorical variable (baseline or 0 weeks, and 10, 20, and 30 weeks after start of treatment) in the linear mixed models (see Multimedia Appendix 2). No differences were found between groups in psychotropic medication use at all assessment periods. Controlling for demographic variables associated with missing data (age and employment status) did not improve model fit in all models (P>.05). Controlling for baseline scores did significantly improve model fit in all models with continuous outcomes (P<.001). Unadjusted (observed) means and outcomes per follow-up assessment are presented in Table 2. Controlling for baseline severity, no group difference was found in decrease of depression severity over time on the IDS-SR30 (overall: b=2.03, 95% CI −1.57 to 5.64; t50.45=1.17, P=.25). Time significantly predicted a decrease of depression severity at all time points (overall: b=−7.13, 95% CI −9.64 to −4.71; t49.55=−5.74, P<.001). For the full sample, the estimated mean depression severity decreased from severe (43.23, 95% CI 42.07-44.40) to moderate (27.1, 95% CI 22.03-32.14) [45,48].

    Controlling for baseline utility scores, time significantly predicted an increase in QALYs gained at all time points (overall, b=.12, 95% CI 0.10-0.13; t73.84= 13.77, P<.001). There was no significant difference in QALY gains between groups (b=−0.01, 95% CI −0.03 to 0.01; t73.74=−0.88, P=.38).

    For reliable change in depression severity (treatment response), a significant association was found between time and treatment response (b=1.81, OR 6.09, 95% CI 0.89-2.89; z=4.00, P<.001), but no significant difference between groups (b=−0.31, OR 0.73, 95% CI −1.61 to 0.99; z =−0.53, P=.60). Twenty patients reported significant improvement at all follow-up assessments (blended CBT: n=10; standard CBT: n=10). There were no patients who consecutively reported deterioration at all follow-up assessments. No deaths occurred during the study. One depression-related, but not treatment-related, serious adverse event occurred in the standard CBT group, in which one person self-harmed. Table 3 provides an overview of reliable change compared with baseline per follow-up, per group.

    The proportion of patients who did not fulfill criteria for a depressive episode at 10, 20, and 30 weeks after the start of treatment did not differ significantly between groups (overall: b=.64, OR 1.89, 95% CI 0.73-4.90; z=1.31, P=.19). Odds of a depressive episode decreased significantly over time (overall: b=−1.44, OR 0.23, 95% CI 0.16-0.35; z=−7.12, P<.001). At 30-week follow-up, 23 of 102 patients (31% of full sample) met criteria for remission, reporting an absence of symptoms for at least 8 weeks. No significant between-group differences were found in the number of comorbid diagnoses. Table 4 provides an overview of the current diagnoses at 30-week follow-up for both groups.

    Table 2. Unadjusted (observed) means and primary clinical and utility outcomes.
    OutcomeBlended CBTaStandard CBTFull sampleBlended CBT vs standard CBT

    		nValuenValuenValueb (95% CI)
    Depression severity (IDS-SRb), mean (SD)





    		Baseline5345.2 (12.1)4941.5 (11.6)10243.4 (12.0)Ref

    		10 weeks4131.9 (12.7)3532.0 (17.5)7632.0 (15.0)−3.36 (−8.20, 1.16)

    		20 weeks3530.7 (16.1)3027.1 (15.7)6529.0 (15.9)1.50 (−6.22, 9.02)

    		30 weeks3629.5 (17.2)2921.1 (15.4)6525.8 (16.9)8.92 (−1.05, 18.50)
    QALYsc (EQ-5D-3L), mean (SD)





    		Baseline5304901020Ref

    		10 weeks410.09 (0.05)350.10 (0.05)760.10 (0.05)−0.01 (−0.04, 0.01)

    		20 weeks350.20 (0.10)300.24 (0.09)650.22 (0.10)−0.02 (−0.06, 0.03)

    		30 weeks360.31 (0.16)290.39 (0.13)650.35 (0.15)−0.03 (−0.10, 0.04)
    Reliable change (IDS-SR), n (%)

    		Baseline5304901020Ref

    		10 weeks4127 (66)3519 (54)7646 (61)0.89 (−1.17, 2.72)

    		20 weeks3521 (60)3019 (63)6540 (62)0.07 (−1.69, 2.01)

    		30 weeks3622 (61)2923 (79)6545 (69)−0.48 (−2.49, 1.47)
    Current depressive episode (MINI-Plus)d, n (%)





    		Baseline5352 (98)4948 (98)102100 (98)Ref

    		10 weeks4427 (61)4118 (44)8545 (44)1.44 (−2.19, 5.07)

    		20 weeks4424 (55)4015 (38)8439 (38)1.30 (−2.35, 5.00)

    		30 weeks3915 (39)3713 (35)7628 (28)0.32 (−3.35, 4.00)

    aCBT: cognitive behavioral therapy.

    bIDS-SR: Inventory of Depressive Symptomatology, self-report version.

    cMINI-Plus: Mini-International Neuropsychiatric Interview Plus.

    dQALYs: quality-adjusted life-years.

    Table 3. Reliable change in depression severity.
    Change at each follow-upBlended CBTaStandard CBTFull sample
    Week 10n=41n=35n=76

    		Deterioration, n (%)1 (2)3 (9)4 (5)

    		No change, n (%)16 (39)16 (48)32 (42)

    		Improvement, n (%)21 (68)10 (29)31 (41)

    		Remission, n (%)3 (7)6 (17)9 (12)
    Week 20n=35n=30n=65

    		Deterioration, n (%)0 (0)0 (0)0 (0)

    		No change, n (%)16 (46)13 (43)29 (45)

    		Improvement, n (%)13 (37)11 (37)24 (37)

    		Remission, n (%)6 (17)6 (20)12 (19)
    Week 30n=36n=29n=65

    		Deterioration, n (%)1 (3)0 (0)1 (2)

    		No change, n (%)15 (42)10 (35)25 (39)

    		Improvement, n (%)14 (40)8 (23)22 (34)

    		Remission, n (%)6 (17)11 (38)17 (26)

    a CBT: cognitive behavioral therapy.

    Table 4. Current DSM-IV-TRa diagnoses at 30-week follow-up.
    DiagnosisBlended CBTb (n=39)Standard CBT (n=37)Full sample (N=76)
    Any comorbidity, n (%)17 (44)18 (49)35 (46)
    Depressive episode, n (%)15 (39)13 (27)28 (37)
    Anxiety, n (%)15 (28)15 (31)30 (29)
    OCDc, n (%)6 (11)3 (6)9 (9)
    PTSDd, n (%)3 (6)2 (4)5 (5)
    Somatoform, n (%)4 (8)4 (8)8 (8)
    Alcohol/drug dependency, n (%)1 (2)3 (6)4 (4)
    Comorbid disorders, mean (SD)1.5 (2.0)1.3 (1.7)1.4 (1.9)

    aDSM-IV: Diagnostic and Statistical Manual of Mental Disorders (Fourth Edition).

    bCBT: Cognitive behavioral therapy.

    cOCD: obsessive compulsive disorder.

    dPTSD: posttraumatic stress disorder.

    Costs

    Before examining cost-effectiveness, costs were explored in separate linear mixed models, controlling for baseline costs. Patients in the blended CBT group reported higher cumulative societal costs on average than patients in standard CBT (overall: b=1410, 95% CI −28.5 to 2776.7; t78.86=2.06, P=.04). Part of this effect appeared to be driven by rare and costly medical events. When costs of hospitalization and in-patient psychiatric care were not included in the societal costs in a sensitivity analysis, the between-group difference was reduced (overall: b=1206, 95% CI −300.9 to 2713.3; t76.99=1.57, P=.12). Based on the estimated marginal means from the mixed-effects model, the overall societal costs for the full sample at 30 weeks were estimated to be mean €10,075 (95% CI €8086-€12,066).

    Regarding average cumulative direct medical costs, both groups reported similar costs (overall: b=55, 95% CI −477.0 to 576.5; t90.19=0.22, P=.83). Average direct medical costs at 30 weeks were estimated to be €4535 (95% CI €3789-€5282). Sensitivity analyses showed that for cumulative indirect nonmedical costs (costs associated with absenteeism and presenteeism), patients in the blended CBT group on average reported higher costs compared with standard CBT (overall: b=140, 95% CI −178.4 to 448.4; t76.22=0.89, P=.37). Average indirect nonmedical costs at 30 weeks were estimated to be €1413 (95% CI €954-€1872). Unadjusted (observed) means and outcomes per follow-up assessment are presented in Table 5.

    Cost-Effectiveness and Cost Utility

    The results from the cost-effectiveness and cost-utility analyses are presented in Table 6. Estimated between-group mean differences in costs, utility, and effects over the full study period were not significant.

    Table 5. Unadjusted mean cumulative costs in Euros.
    CostsBlended CBTStandard CBTFull sampleBlended CBT vs standard CBT

    		nMean (SD)nMean (SD)nMean (SD)b (95% CI)
    Societal costs (€)







    		Baseline532205 (2615)491401 (1761)1021819 (891)Ref

    		10 weeks405140 (5229)352996 (2608)754140 (2419)1482 (−114, 2989)

    		20 weeks339523 (9466)285765 (4974)617798 (5202)3137 (288, 5980)

    		30 weeks3012,401 (13,198)248493 (7239)5410,664 (6316)3923 (−148, 7963)
    Direct medical costs (€)

    		Baseline53628 (888)49411 (389)102524 (381)Ref

    		10 weeks402363 (2743)351697 (934)752052 (1661)322 (−257, 920)

    		20 weeks333834 (3833)283025 (1846)613463 (2610)308 (−734, 1343)

    		30 weeks304728 (4504)244.287 (3743)544527 (3398)25 (−1525, 1512)

    aCBT: Cognitive behavioral therapy.

    Table 6. Results of the cost-effectiveness and cost-utility analyses.
    OutcomeBlended CBTa vs standard CBTICERbCost-effectiveness plane distributionc (%)

    		Difference cost, mean difference (95% CI)Difference effect, mean or risk differenced (95% CI)
    		NWeSWfNEgSEh
    Societal perspective






    		RCIi1183 (−399, 2765)0.03 (−.010, .15)39,43310.0—89.20.7

    		MINIj1183 (−399, 2765)0.06 (−0.05, 0.19)19,71689.20.710.0—

    		QALYk1183 (−399, 2765)0.01 (−0.03, 0.04)185,88034.40.664.90.2
    Health care provider perspective






    		RCI−176 (−659, 343)0.03 (−0.10, 0.15)−58675.24.920.269.7

    		MINI−176 (−659, 343)0.06 (−0.05, 0.19)−293320.269.75.24.9

    		QALY−176 (−659, 343)0.01 (−0.03, 0.04)−29,3333.031.922.342.7

    aCBT: Cognitive behavioral therapy.

    bICER: incremental cost-effectiveness ratio.

    cFor plane distribution, NW=more expensive, less effective; SW=less expensive, less effective; NE=more expensive, more effective; SE=less expensive, more effective.

    dRisk difference: RCI and MINI; Mean: QALY.

    eNW: northwest.

    fSW: southwest.

    gNE: northeast.

    hSE: southeast.

    iRCI: reliable change index (based on IDS-SR).

    jMINI: Mini-International Neuropsychiatric Interview Plus (diagnostic interview diagnosis of a depressive episode).

    kQALY: quality-adjusted life-year (based on EQ-5D-3L).

    Societal Perspective

    For response to treatment, the ICER was 39,433, meaning that an additional treatment response in blended CBT was associated with €39,433 higher costs compared with standard CBT. Of the estimated cost-effect pairs, 89.2% was located in the northeast quadrant (more effective and more expensive), 10% in the northwest quadrant (less effective and more expensive), and 0.7% in the southeast quadrant (more effective and less expensive). The probability of blended CBT being cost-effective compared with standard CBT was 0.01 at a ceiling ratio of €0 per additional response to treatment, and 0.02, 0.18, and 0.28 at ceiling ratios of €5000, €15,000, and €20,000 per response to treatment, respectively. The probability of blended CBT being cost-effective compared with standard CBT rose to 0.37 at a ceiling ratio of €25,000 per additional response to treatment. The cost-effectiveness plane and cost-effectiveness acceptability curve are presented in Figure 2.

    For occurrence of depressive episodes, the ICER was 19,716. This means that avoiding an additional depressive episode in blended CBT was associated with €19,716 higher costs compared with standard CBT. The probability of blended CBT being cost-effective compared with standard CBT was 0.01 at a ceiling ratio of €0 per depressive episode. This was the highest possible probability.

    For QALYs, the ICER was 185,880, meaning that one extra QALY in blended CBT was associated with €185,880 higher costs compared with standard CBT. The probability of blended CBT being cost-effective compared with standard CBT was 0.01 at a ceiling ratio of €0 per QALY. The probability of blended CBT being cost-effective compared with standard CBT rose to 0.02 at a ceiling ratio of €25,000 per extra QALY.

    ‎
    Figure 2. Cost-effectiveness planes for response to treatment (blended CBT versus standard CBT) from the societal perspective (top left) and health care provider perspective (top right), and cost-effectiveness acceptability curves for response to treatment (blended CBT versus standard CBT) from the societal perspective (bottom left) and health care provider perspective (bottom right). CBT: cognitive behavioral therapy
    View this figure
    Health Care Provider Perspective

    For response to treatment, the ICER was −5867, meaning that an additional treatment response in blended CBT was associated with €5867 lower costs compared with standard CBT. Of the estimated cost-effect pairs, 69.7% was located in the southeast quadrant (more effective and less expensive), 20.2% in the northeast quadrant (more effective and more expensive), 5.2% in the northwest quadrant (less effective and more expensive), and 4.9% in the southwest quadrant (less effective, less expensive). The probability of blended CBT being cost-effective compared with standard CBT was 0.75 at a ceiling ratio of €0 per additional response to treatment, and 0.80 at a ceiling ratio of €1000 per additional response to treatment. The probability of blended CBT being cost-effective compared with standard CBT rose to 0.88 at a ceiling ratio of €5000 per additional response to treatment. The cost-effectiveness plane and cost-effectiveness acceptability curve are presented in Figure 2.

    For occurrence of depressive episodes, the ICER was −2933. This means that an additional depressive episode in blended CBT was associated with €2933 lower costs compared with standard CBT. The probability of blended CBT being cost-effective compared with standard CBT was 0.75 at a ceiling ratio of €0 per depressive episode.

    For QALYs, the ICER was −29,333, meaning that one extra QALY in blended CBT was associated with €29,333 lower costs compared with standard CBT. The probability of blended CBT being cost-effective compared with standard CBT was 0.75 at a ceiling ratio of €0 per QALY gained, and 0.81 at €5000 per QALY gained. The probability of blended CBT being cost-effective compared with standard CBT rose to 0.85 at €10,000 per QALY gained.


    Overview

    This pilot RCT focused on depression treatment in Dutch routine specialized mental health care and compared the costs and effects of blended CBT for MDD to standard (evidence-based) face-to-face CBT over a 30-week time frame. The study examined whether blended CBT has the potential to lead to clinical effects that are comparable to the effects of standard CBT, at lower costs. To the best of our knowledge, our study was the first to explore the cost-effectiveness of integrated (rather than add-on) blended depression treatment in routine mental health care and compare this blended treatment to existing, evidence-based, face-to-face CBT.

    Principal Findings

    The results suggest that blended CBT for depression has the potential to lead to costs and clinical effects that are comparable to the costs and effects associated with standard CBT. In both treatment groups, the severity of depressive symptoms decreased, the probability of having a depression diagnosis lessened, and quality of life improved. When costs and effects were combined, results were mixed. From a societal perspective, which includes productivity losses, blended CBT was not cost-effective compared with standard CBT. From a health care provider perspective, blended CBT had an acceptable probability of being cost-effective compared with standard CBT for treatment response and QALYs (cost utility), but not for depression diagnosis (depressive episodes).

    Results show that blended CBT patients received significantly fewer face-to-face sessions than standard CBT patients (10 versus 13 sessions). In line with our expectations, blended CBT also led to shorter treatment duration (19 versus 33 weeks). After 20 weeks of treatment, 51% (27 of 53) of blended CBT patients completed 75% or more of the treatment protocol (at least 14 of 18 sessions in total) versus 5% in the CBT group (3 of 49). After 30 weeks, the percentage of treatment completers rose to 85% for blended CBT and 45% for standard CBT. Combined with therapist time required for online feedback, blended CBT and standard CBT unexpectedly required similar per-patient time investments from therapists (14 versus 13 hours). This was mainly driven by the mean number of sessions in standard CBT, which was considerably lower than expected based on the protocol (13 versus 18 sessions).

    Comparison with Other Work

    Because this is one of the first studies to compare both costs and effects of blended depression treatment to treatment as usual, the possibility of comparing our findings to those of previous research is limited. The clinical findings appear to be in line with other studies that examined the clinical effectiveness of combined online and face-to-face treatment for depression. However, in this study we defined blended treatment as an integrated approach, replacing face-to-face sessions with online modules rather than a combined approach that adds an online intervention to a face-to-face CBT protocol. For example, Berger and colleagues [33] evaluated the clinical effectiveness of an evidence-based unguided CBT self-help intervention (Deprexis) combined with regular psychotherapy in outpatient specialized depression care compared with regular psychotherapy. The results of their RCT (N=98) showed superiority of the combined treatment [33] over regular treatment for reduced depressive symptoms (via Beck Depression Inventory-II, BDI-II) (Cohen d=.51) at posttreatment (12 weeks). In another RCT (N=229), this unguided self-help intervention was added to a psychodynamic psychotherapeutic treatment (treatment as usual) for depressed inpatients and compared with online information plus treatment as usual [35]. After treatment, treatment as usual plus unguided self-help was superior to treatment as usual plus active control (Cohen d=.44).

    A naturalistic study in routine specialized mental health care by Kenter and colleagues [36] showed comparable clinical effects for blended and standard face-to-face treatment for depressed patients, as measured with Global Assessment of Functioning (GAF) scores (N=3175) [36]. In this study, the combined rather than integrated approach led to longer treatment duration and more treatment sessions (combined total of online and face-to-face sessions), making blended treatment by design more costly than face-to-face treatment [36].

    Offering blended treatment in a group format rather than to patients individually could potentially further lower costs associated with treatment. Schuster and colleagues [64] examined change in severity of depressive symptoms (via Center for Epidemiologic Studies Depression Scale, CES-D) in a combined blended group therapy compared with a waitlist control group. Participants (N=46) were adults with depressive symptoms who were recruited from the general population in Austria. Blended therapy consisted of eight 90-minute group sessions, combined with access to an e-learning platform. Treatment was provided at the University of Salzburg. The group intervention was superior to waitlist (Cohen d=.87) after treatment.

    Finally, in a recent study by Thase and colleagues [25], an integrated approach was chosen for treating medication-free adults with MDD (N=154). Treatment was provided in two university clinics. In the blended treatment, nine online modules were combined with 12 face-to-face sessions of 25 minutes. Compared with 20 CBT sessions of 50 minutes, blended CBT led to noninferior results (Cohen d=.05) on the HAM-D after 16 weeks of treatment.

    Limitations

    There are a number of factors that need to be considered when interpreting the findings. First, the sample size was smaller than anticipated, which led to less stable probability estimates. This was mainly due to time constraints of conducting the study. The achieved sample is considered adequate for the central aim of the study, which was to guide further development and research in blended CBT. Future studies should focus on hypothesis testing in larger samples, using a noninferiority design.

    Second, within this study, patients could be followed for 30 weeks. Although this was considered an acceptable time period to assess changes in outcomes during and shortly after treatment, future studies could include a long-term follow-up (eg, 1 year after treatment). This would provide more insight into the course of depression, treatment trajectories, quality of life, and costs over time. When examining costs, future studies could also include specific costs associated with delivering the interventions, such as the hosting and maintenance of the online treatment platform. Unfortunately, these data were not available in this study.

    Third, this sample was characterized by a high level of heterogeneity, both in clinical and cost characteristics. On the one hand, this is a positive aspect because it is representative of the population in routine specialized mental health care. On the other hand, it led to some baseline imbalances between groups. This is most apparent in the average direct medical and nonmedical costs patients reported in the month before baseline, which were higher in the blended CBT group than the standard CBT group. This was primarily caused by the fact that three patients in the blended group reported having been admitted to an in-patient psychiatric ward before baseline assessment versus no patients in the standard CBT group.

    The imbalances at baseline increased the uncertainty around the cost and effectiveness estimates, especially at the 30-week follow-up. This is reflected in the analyses concerning treatment response, quality of life, and direct medical costs (health care provider perspective). The models that included time were in favor of the standard CBT group, whereas the models that did not include time were in favor of blended CBT. It is important to keep in mind that all findings were nonsignificant and varied around zero, signifying no differences between groups.

    Finally, it should be noted that in the study protocol [41] more outcomes were included than could be reported on in this paper, such as working alliance and depressive cognitions. Information on these outcomes will be provided in forthcoming publications. Next, in the study protocol, it was stated that treatment response would be assessed at 30 weeks. This might lead readers to wonder whether there was a deviation from the planned analyses. However, within the cost-effectiveness framework, the focus lies on cumulative outcomes. Therefore, cumulative group differences were estimated with linear mixed models over the full study period, rather than estimating a group difference at a single time point (30 weeks).

    Conclusion

    This study is one of the first to examine both costs and effects of integrated blended CBT for depression in specialized mental health care. The results are promising and suggest it is feasible to digitalize part of the therapist-patient interaction, even in the complex patient population that characterizes specialized mental health care. Blended CBT appears to lead to comparable clinical effects as standard CBT, may increase treatment adherence, and could potentially speed up patient flow through the treatment process. From a societal perspective, blended CBT is not considered cost-effective compared with standard CBT. However, there is an acceptable probability that blended CBT is cost-effective from the perspective of the health care provider. Further research in a larger sample seems warranted, which should focus on optimizing the clinical effects of blended treatment as well as the cost-impact.

    Acknowledgments

    This study was funded by ZonMw (project number 837001007). The authors gratefully acknowledge the contribution of all participants, research assistants, therapists, and all others who contributed to the study. We would like to thank Adriaan Hoogendoorn and Judith Bosmans for their invaluable advice during the data-analysis process.

    Authors' Contributions

    HR and PvO obtained funding for the study. All authors contributed to the design of the study. HR, PvO, JEW, LCK, and JR developed the intervention. LCK coordinated the recruitment of patients and the data collection. HR, FS, JL, KN, JR, and LK developed the analytic plan. FS, JL, KN, and LCK analyzed the data. LCK wrote the manuscript. All authors read and revised the manuscript and gave final approval of the version to be published. JR passed away in July 2019, shortly before the manuscript was submitted for the final round of the peer-review process.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Cost categories and prices (€, 2014).

    PDF File (Adobe PDF File), 35 KB
    ‎
    Multimedia Appendix 2

    Compliance with questionnaires at follow-up.

    PDF File (Adobe PDF File), 21 KB
    ‎
    Multimedia Appendix 3

    CONSORT-EHEALTH checklist (V 1.6.1).

    PDF File (Adobe PDF File), 2515 KB
    1. Cuijpers P, Huibers M, Ebert DD, Koole SL, Andersson G. How much psychotherapy is needed to treat depression? A metaregression analysis. J Affect Disord 2013;149(1-3):1-13. [CrossRef] [Medline]
    2. Karyotaki E, Smit Y, Holdt Henningsen K, Huibers MJ, Robays J, de Beurs D, et al. Combining pharmacotherapy and psychotherapy or monotherapy for major depression? A meta-analysis on the long-term effects. J Affect Disord 2016 Apr;194:144-152. [CrossRef] [Medline]
    3. Butler AC, Chapman JE, Forman EM, Beck AT. The empirical status of cognitive-behavioral therapy: a review of meta-analyses. Clin Psychol Rev 2006 Jan;26(1):17-31. [CrossRef] [Medline]
    4. Kohn R, Saxena S, Levav I, Saraceno B. The treatment gap in mental health care. Bull World Health Organ 2004 Nov;82(11):858-866 [FREE Full text] [Medline]
    5. Bremmer F, van Es M. Een analyse van de verwachte kosten en baten van eHealth-blended behandelen en begeleiden. Amersfoort, the Netherlands: GGZ Nederland; 2013.
    6. Araya R, Zitko P, Markkula N, Rai D, Jones K. Determinants of access to health care for depression in 49 countries: a multilevel analysis. J Affect Disord 2018 Jul;234:80-88. [CrossRef] [Medline]
    7. Health Quality Ontario. Psychotherapy for major depressive disorder and generalized anxiety disorder: a health technology assessment. Ont Health Technol Assess Ser 2017;17(15):1-167. [Medline]
    8. Andersson G. The promise and pitfalls of the internet for cognitive behavioral therapy. BMC Med 2010;8:82 [FREE Full text] [CrossRef] [Medline]
    9. Ruwaard J, Kok RN. Wild West eHealth: Time to hold our horses? Eur Health Psychol 2015;17(1):45-49.
    10. Emmelkamp PM, David D, Beckers T, Muris P, Cuijpers P, Lutz W, et al. Advancing psychotherapy and evidence-based psychological interventions. Int J Methods Psychiatr Res 2014;23 Suppl 1:58-91. [CrossRef] [Medline]
    11. Andrews G, Basu A, Cuijpers P, Craske M, McEvoy P, English C, et al. Computer therapy for the anxiety and depression disorders is effective, acceptable and practical health care: An updated meta-analysis. J Anxiety Disord 2018 Apr;55:70-78 [FREE Full text] [CrossRef] [Medline]
    12. Wright JH, Wright AS, Albano AM, Basco MR, Goldsmith LJ, Raffield T, et al. Computer-assisted cognitive therapy for depression: maintaining efficacy while reducing therapist time. Am J Psychiatry 2005 Jun;162(6):1158-1164. [CrossRef] [Medline]
    13. Richards D, Richardson T. Computer-based psychological treatments for depression: a systematic review and meta-analysis. Clin Psychol Rev 2012 Jun;32(4):329-342. [CrossRef] [Medline]
    14. Andersson G, Cuijpers P. Internet-based and other computerized psychological treatments for adult depression: a meta-analysis. Cogn Behav Ther 2009;38(4):196-205. [CrossRef] [Medline]
    15. Andersson G, Rozental A, Shafran R, Carlbring P. Long-term effects of internet-supported cognitive behaviour therapy. Expert Rev Neurother 2018 Jan;18(1):21-28. [CrossRef] [Medline]
    16. Webb CA, Rosso IM, Rauch SL. Internet-based cognitive-behavioral therapy for depression: current progress and future directions. Harv Rev Psychiatry 2017;25(3):114-122 [FREE Full text] [CrossRef] [Medline]
    17. Carlbring P, Andersson G, Cuijpers P, Riper H, Hedman-Lagerlöf E. Internet-based vs face-to-face cognitive behavior therapy for psychiatric and somatic disorders: An updated systematic review and meta-analysis. Cogn Behav Ther 2018;47(1):1-18. [CrossRef] [Medline]
    18. Kolovos S, van Dongen JM, Riper H, Buntrock C, Cuijpers P, Ebert DD, et al. Cost effectiveness of guided Internet-based interventions for depression in comparison with control conditions: an individual-participant data meta-analysis. Depress Anxiety 2018;35(3):209-219 [FREE Full text] [CrossRef] [Medline]
    19. Cuijpers P, Riper H, Andersson G. Internet-based treatment of depression. Curr Opin Psychol 2015;4(1):131-135 [FREE Full text] [CrossRef]
    20. Schueller SM, Tomasino KN, Mohr DC. Integrating human support into behavioral intervention technologies: the efficiency model of support. Clin Psychol Sci Pract 2016;24(1):27-45. [CrossRef]
    21. Wentzel J, van der Vaart R, Bohlmeijer ET, van Gemert-Pijnen JE. Mixing online and face-to-face therapy: how to benefit from blended care in mental health care. JMIR Ment Health 2016;3(1):e9 [FREE Full text] [CrossRef] [Medline]
    22. Kooistra LC, Ruwaard J, Wiersma JE, van Oppen P, van der Vaart R, van Gemert-Pijnen JE, et al. Development and initial evaluation of blended cognitive behavioural treatment for major depression in routine specialized mental health care. Internet Interv 2016 May;4:61-71 [FREE Full text] [CrossRef] [Medline]
    23. Erbe D, Eichert HC, Riper H, Ebert DD. Blending face-to-face and Internet-based interventions for the treatment of mental disorders in adults: systematic review. J Med Internet Res 2017;19(9):e306 [FREE Full text] [CrossRef] [Medline]
    24. Andersson G, Cuijpers P. Pros and cons of online cognitive-behavioural therapy. Br J Psychiatry 2008 Oct;193(4):270-271. [CrossRef] [Medline]
    25. Thase ME, Wright JH, Eells TD, Barrett MS, Wisniewski SR, Balasubramani GK, et al. Improving the efficiency of psychotherapy for depression: computer-assisted versus standard CBT. Am J Psychiatry 2018;175(3):242-250. [CrossRef] [Medline]
    26. Kleiboer A, Smit J, Bosmans J, Ruwaard J, Andersson G, Topooco N, et al. European COMPARative Effectiveness research on blended depression treatment versus treatment-as-usual (E-COMPARED): study protocol for a randomized controlled, non-inferiority trial in eight European countries. Trials 2016;17(1):387 [FREE Full text] [CrossRef] [Medline]
    27. Romijn G, Riper H, Kok R, Donker T, Goorden M, Hakkaart-van Roijen L, et al. Cost-effectiveness of blended vs face-to-face cognitive behavioural therapy for severe anxiety disorders: study protocol of a randomized controlled trial. BMC Psychiatry 2015;15:311 [FREE Full text] [CrossRef] [Medline]
    28. Vis C, Kleiboer A, Prior R, Bønes E, Cavallo M, Clark S, et al. Implementing and up-scaling evidence-based eMental health in Europe: the study protocol for the MasterMind project. Internet Interventions 2015;2(4):399-409. [CrossRef]
    29. Mol M, Dozeman E, van Schaik DJ, Vis CP, Riper H, Smit JH. The therapist's role in the implementation of internet-based cognitive behavioural therapy for patients with depression: study protocol. BMC Psychiatry 2016;16(1):338 [FREE Full text] [CrossRef] [Medline]
    30. Hickie IB, Davenport TA, Luscombe GM, Moore M, Griffiths KM, Christensen H. Practitioner-supported delivery of internet-based cognitive behaviour therapy: evaluation of the feasibility of conducting a cluster randomised trial. Med J Aust 2010;192(11 Suppl):S31-S35. [Medline]
    31. Høifødt RS, Lillevoll KR, Griffiths KM, Wilsgaard T, Eisemann M, Waterloo K, et al. The clinical effectiveness of web-based cognitive behavioral therapy with face-to-face therapist support for depressed primary care patients: randomized controlled trial. J Med Internet Res 2013;15(8):e153 [FREE Full text] [CrossRef] [Medline]
    32. Kessler D, Lewis G, Kaur S, Wiles N, King M, Weich S, et al. Therapist-delivered Internet psychotherapy for depression in primary care: a randomised controlled trial. Lancet 2009 Aug 22;374(9690):628-634. [CrossRef] [Medline]
    33. Berger T, Krieger T, Sude K, Meyer B, Maercker A. Evaluating an e-mental health program. J Affect Disord 2018;227:455-462. [CrossRef] [Medline]
    34. Meyer B, Berger T, Caspar F, Beevers CG, Andersson G, Weiss M. Effectiveness of a novel integrative online treatment for depression (Deprexis): randomized controlled trial. J Med Internet Res 2009;11(2):e15 [FREE Full text] [CrossRef] [Medline]
    35. Zwerenz R, Becker J, Knickenberg RJ, Siepmann M, Hagen K, Beutel ME. Online self-help as an add-on to inpatient psychotherapy: efficacy of a new blended treatment approach. Psychother Psychosom 2017;86(6):341-350. [CrossRef] [Medline]
    36. Kenter RM, van de Ven PM, Cuijpers P, Koole G, Niamat S, Gerrits RS, et al. Costs and effects of Internet cognitive behavioral treatment blended with face-to-face treatment: results from a naturalistic study. Internet Interventions 2015;2(1):77-83. [CrossRef]
    37. American Psychiatric Association. Diagnostic and Statistical Manual of Mental Disorders, 4th Edition, Text Revision (DSM-IV-TR). Washington, DC: American Psychiatric Publishing; 2000.
    38. Spijker J, Bockting CL, Meeuwissen JA, van Vliet IM, Emmelkamp PM, Hermens ML, et al. Multidisciplinaire richtlijn depressie (derde revisie). Richtlijn voor de diagnostiek, behandeling en begeleiding van volwassen patiënten met een depressieve stoornis. Utrecht, the Netherlands: Trimbos-instituut; 2013.
    39. Sheehan DV, Lecrubier Y, Sheehan KH, Amorim P, Janavs J, Weiller E, et al. The Mini-International Neuropsychiatric Interview (M.I.N.I.): the development and validation of a structured diagnostic psychiatric interview for DSM-IV and ICD-10. J Clin Psychiatry 1998;59 Suppl 20:22-33;quiz 34. [Medline]
    40. van Vliet IM, de Beurs E. Het Mini Internationaal Neuropsychiatrisch Interview (MINI): een kort gestructureerd diagnostisch psychiatrisch interview voor DSM-IV en ICD-10-stoornissen. Tijdschrift voor de Psychiatrie 2007;49(6):197-393.
    41. Kooistra LC, Wiersma JE, Ruwaard J, van Oppen P, Smit F, Lokkerbol J, et al. Blended vs. face-to-face cognitive behavioural treatment for major depression in specialized mental health care: study protocol of a randomized controlled cost-effectiveness trial. BMC Psychiatry 2014;14:290 [FREE Full text] [CrossRef] [Medline]
    42. Bockting CL, Huibers MJ. Protocollaire behandeling van patienten met een depressieve stoornis. In: Keijsers GPJ GJ, van Minnen A, Hoogduin CA, editors. Protocollaire behandelingen voor volwassenen met psychische klachten deel 1. Amsterdam, the Netherlands: Uitgeverij Boom; 2011:251-288.
    43. Beck JS. Cognitive Behavior Therapy: Basics and Beyond. New York: Guilford Press; 2011.
    44. Beck AT, Rush AJ, Shaw BF, Emery G. Cognitive Therapy of Depression. New York: Guilford Press; 1980.
    45. Rush AJ, Trivedi MH, Ibrahim HM, Carmody TJ, Arnow B, Klein DN, et al. The 16-Item Quick Inventory of Depressive Symptomatology (QIDS), clinician rating (QIDS-C), and self-report (QIDS-SR): a psychometric evaluation in patients with chronic major depression. Biol Psychiatry 2003 Sep 01;54(5):573-583. [Medline]
    46. Rush AJ, Carmody T, Reimitz P. The Inventory of Depressive Symptomatology (IDS): clinician (IDS‐C) and self‐report (IDS‐SR) ratings of depressive symptoms. Int J Method Psychiat Res 2000;9(2):45-59. [CrossRef]
    47. Cusin C, Yang H, Yeung A, Fava M. Rating scales for depression. In: Bear L, Blais MA, editors. Handbook of Clinical Rating Scales and Assessment in Psychiatry and Mental Health. Totowa, NJ: Humana Press; 2019.
    48. Rush AJ, Gullion CM, Basco MR, Jarrett RB, Trivedi MH. The Inventory of Depressive Symptomatology (IDS): psychometric properties. Psychol Med 1996 May;26(3):477-486. [Medline]
    49. Trivedi MH, Rush AJ, Ibrahim HM, Carmody TJ, Biggs MM, Suppes T, et al. The Inventory of Depressive Symptomatology, Clinician Rating (IDS-C) and Self-Report (IDS-SR), and the Quick Inventory of Depressive Symptomatology, Clinician Rating (QIDS-C) and Self-Report (QIDS-SR) in public sector patients with mood disorders: a psychometric evaluation. Psychol Med 2004;34(1):73-82. [CrossRef] [Medline]
    50. Jacobson NS, Truax P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. In: Kazdin AE, editor. Methodological Issues and Strategies in Clinical Research. Washington, DC: American Psychological Association; Feb 1991:631-648.
    51. Trivedi MH, Rush AJ, Wisniewski SR, Nierenberg AA, Warden D, Ritz L, STAR*D study team. Evaluation of outcomes with citalopram for depression using measurement-based care in STAR*D: implications for clinical practice. Am J Psychiatry 2006 Jan;163(1):28-40. [CrossRef] [Medline]
    52. EuroQol Group. EuroQol--a new facility for the measurement of health-related quality of life. Health Policy 1990 Dec;16(3):199-208. [Medline]
    53. Lamers LM, McDonnell J, Stalmeier PF, Krabbe PF, Busschbach JJ. The Dutch tariff: results and arguments for an effective design for national EQ-5D valuation studies. Health Econ 2006;15(10):1121-1132. [CrossRef] [Medline]
    54. Hakkaart-van Roijen L, van Straten A, Tiemens B, Donker M. Trimbos/iMTA questionnaire for costs associated with psychiatric illness (TIC-P). Rotterdam, the Netherlands: Institute of Medical Technology Assessment (iMTA); 2002.
    55. Hakkaart-van Roijen L, van der Linden N, Bouwmans C, Kanters T, Tan S. Kostenhandleiding: methodologie van kostenonderzoek en referentieprijzen voor economische evaluaties in de gezondheidszorg. Rotterdam, the Netherlands: Institute of Medical Technology Assessment (iMTA); 2016.
    56. Hakkaart-Van Roijen L, Tan SS, Bouwmans CA. Handleiding voor kostenonderzoek--methoden en standaard kostprijzen voor economische evaluaties in de gezondheidszorg. Rotterdam, the Netherlands: Instituut voor Medical Technology Assessment (iMTA), Erasmus Universiteit Rotterdam; 2010.
    57. Z-index. The Hague, the Netherlands: Zorginstituut Nederland; 2016.
    58. Hesser H. Modeling individual differences in randomized experiments using growth models: recommendations for design, statistical analysis and reporting of results of internet interventions. Internet Interventions 2015 May;2(2):110-120. [CrossRef]
    59. White IR, Carpenter J, Horton NJ. Including all individuals is not enough: lessons for intention-to-treat analysis. Clinical Trials 2012 Jul 02;9(4):396-407. [CrossRef]
    60. Bates D, Mächler M, Bolker B, Walker S. Fitting linear mixed-effects models using lme4. J Stat Soft 2015;67(1):1-48. [CrossRef]
    61. R Core Team. R: a language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing; 2018.
    62. Fenwick E, O'Brien BJ, Briggs A. Cost-effectiveness acceptability curves--facts, fallacies and frequently asked questions. Health Econ 2004;13(5):405-415. [CrossRef] [Medline]
    63. Hakkaart-van Roijen L, van Straten A, Maiwenn A, Rutten F, Donker M. Cost-utility of brief psychological treatment for depression and anxiety. Br J Psychiatry 2006;188:323-329. [CrossRef] [Medline]
    64. Schuster R, Leitner I, Carlbring P, Laireiter A. Exploring blended group interventions for depression: randomised controlled feasibility study of a blended computer- and multimedia-supported psychoeducational group intervention for adults with depressive symptoms. Internet Interv 2017;8:63-71 [FREE Full text] [CrossRef] [Medline]

    ‎
    CBT: cognitive behavioral therapy
    ICER: Incremental Cost-Effectiveness Ratio
    IDS-SR: Inventory of Depressive Symptomatology Self-Report
    MDD: major depressive disorder
    MINI-Plus: Mini-International Neuropsychiatric Interview Plus
    QALY: quality-adjusted life-year
    RCI: reliable change index
    TiC-P: Trimbos and iMTA questionnaire on Costs associated with Psychiatric illness

    Edited by G Eysenbach; submitted 04.04.19; peer-reviewed by S Kolovos, R Schuster; comments to author 11.05.19; revised version received 25.07.19; accepted 18.08.19; published 29.10.19

    Copyright

    ©Lisa Catharine Kooistra, Jenneke Elize Wiersma, Jeroen Ruwaard, Koen Neijenhuijs, Joran Lokkerbol, Patricia van Oppen, Filip Smit, Heleen Riper. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 29.10.2019.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.