Evaluation of Two Web-Based Interventions (REMOTION and Res-Up!) for Clients From Psychotherapy Waitlists in Routine Outpatient Psychotherapy (Therapy Online Plus-TOP): Randomized Controlled Trial

Introduction

Background

Approximately 1 (17.6%) in 6 people worldwide met the criteria for a mental disorder, and nearly one-third (29.2%) had experienced a mental disorder at some point in their lives [1]. Mental health concerns and disorders have a profound negative impact on individuals affected by symptoms and, therefore, on society as a whole. In Germany, despite clinical need, only 11% to 40% of patients receive appropriate treatment depending on various factors, indicating substantial gaps in psychotherapy care access and contribution [2,3]. The most common reasons are long waiting times for psychotherapy (eg, 20.1 weeks in 2019 [4]), fear of stigmatization, or barriers inherent to the type of disorder (eg, social anxiety and suicidality [5] or depression [6]).

Interventions provided via apps or web are an innovative way of delivering much-needed treatments and can thereby help in closing the earlier mentioned psychotherapy care gap [7-9]. With limited access to psychotherapy, online interventions are an economical, flexible, and practical addition to conventional treatment [8]. Particularly, when face-to-face psychotherapy is not available due to various types of barriers (such as insufficient availability of therapists or fear of facing stigmatization), internet-based interventions may help to avoid long waiting times for treatment and thus mitigate the chronification of mental health problems [7]. Research on internet-based self-help interventions in psychotherapy demonstrates the effectiveness of different intervention programs for a variety of mental disorders [8,10,11]. Different types of internet-based interventions can function as stand-alone online programs or be conceptualized as blended treatments—the combination of face-to-face psychotherapy with online interventions [12].

Concerning internet-based interventions during waiting times for psychotherapy, a recent review of Huang et al [13] found only 5 randomized controlled trials (RCTs) and 3 uncontrolled studies focusing on depression and anxiety. While these studies showed no additional benefit for internet-based interventions compared to waiting lists, the authors concluded that more high-quality RCTs with larger sample sizes are warranted. Some studies, however, show incremental benefits for internet-based psychotherapy over waiting controls with varying effect sizes [14-17]. In particular, transdiagnostic internet-based interventions might be a promising approach as they are suitable for patients with a wide spectrum of mental health problems and comorbid disorders can be addressed [16,18]. Furthermore, universally tailored transdiagnostic treatments may be more effective than tailored ones [19]. When looking at psychotherapy interventions in general, a differentiation between interventions focusing more on compensation and those focusing more on capitalization is possible. Compensation-oriented interventions identify dysfunctional maintenance factors of psychopathology and teach or train the person to use new strategies or build new behavior against relative deficits, whereas capitalization-oriented interventions aim to draw on the person’s already existing strengths (eg, action repertoire, resilience strategies, and external as well as internal resources) [20-24]. Research indicates that focusing on strength-based approaches (including capitalization) in psychotherapies is beneficial for therapy outcomes, especially with regard to well-being [25-27]. Compensation-oriented interventions might be needed during waiting times to counter dysfunctional maintenance factors of psychopathology, but when therapeutic support is limited, it might be particularly important to activate and support a person’s strengths and thus follow a capitalization-oriented approach. Research on compensation and capitalization interventions, especially including control groups (CGs), is difficult and sparse [20,23,28-31]. To increase knowledge concerning said concepts, we set the emotion regulation intervention as more compensation focused and the resilience improvement intervention as more capitalization focused, as both concepts have been shown to be effective methods in psychotherapy [32].

Thus, this RCT explores, for the first time, the effect of 2 transdiagnostic web-based interventions delivered via an internet browser compared to a care-as-usual–only CG in routine outpatient care.

Emotion Regulation (as the More Compensation-Oriented Approach)

The concept of emotion regulation is relevant to the treatment of a wide range of mental health disorders [33-37]. Different definitions of emotion regulation exist in the literature. According to Gross [38], emotion regulation is the process by which individuals influence which emotions they have, when they have them, and how they experience and express them. Several web- and mobile-based mental health interventions have been developed to target this construct [39,40]. Using the extended process model of emotion regulation that differentiates between the identification, selection, and implementation phases of emotion regulation as a framework [41], the web-based REMOTION intervention was developed [42]. The internet-based program used in this trial aims to improve patients’ emotion regulation and thus reduce their symptom severity. The program has been tested in an outpatient psychotherapy setting in a pilot RCT [43] and also as an add-on to acute inpatient psychiatric care in a pilot RCT [44,45].

Resilience (as the more Capitalization-Oriented Approach)

Resilience and resources or strengths are often discussed in the field of positive psychology [46], focusing on factors increasing health rather than decreasing symptoms. Most research on resilience aims at building new strengths [47]. However, the evidence-based personal model of resilience (PMR), developed by Padesky and Mooney [48] from a cognitive perspective, takes a clear capitalization approach: The PMR emphasizes individuals’ preexisting strengths and resilient strategies, aiming to enhance awareness and the use of these strengths. Its basic premise is that everyone possesses resilience strategies that are already part of their behavioral repertoire and can be used in challenging situations. The PMR is grounded in evidence-based standards for cognitive behavioral therapy [49] and has proven to be helpful in several studies [50-52]. In this study, Res-Up! (Resilienz – unkompliziert und persönlich!) is implemented as a web-based version of the program to foster resilience in participants. The internet-based version of the PMR “Res-Up!” (resilience program of the University of Witten/Herdecke) as a stand-alone program without a CG showed significant improvements over time, with medium effect sizes in improving resilience (d=0.51-0.55) and emotional competence (d=0.51), greater effect sizes for self-compassion (d=0.70), and small effect sizes for self-esteem (d=0.41) [53].

Aims

Many controlled studies have shown the efficacy of internet-based interventions. There is still limited evidence about the effectiveness of transdiagnostic approaches for internet-based interventions in routine psychotherapeutic practice and specifically their usefulness in bridging the gap between no support and the beginning of psychotherapy, especially during waiting times for therapy [13]. In this study, the goal was to assess the effectiveness of an emotion regulation improving intervention (REMOTION, more compensation oriented) and a resilience improving intervention (Res-Up!, more capitalization oriented) web-based intervention in an outpatient psychotherapeutic routine setting in comparison to a care-as-usual–only CG. In accordance with previous research on internet-based interventions as stand-alone, blended care and compared to care-as-usual and waiting conditions [14,15,54,55], we hypothesized that both interventions were more efficacious than no additional treatment. The primary outcome targets overall symptom severity. The secondary outcomes assess improvement of emotion regulation and resilience as the specific target areas of the individual interventions, whereas other outcomes target depressive symptoms, self-esteem, and self-compassion.

Methods

Study Design

In this 3-arm multicenter RCT, 421 participants were randomized into 1 of 2 web-based interventions (“REMOTION” and “Res-Up!”) or a CG. Three assessments were conducted: baseline, after 6 weeks (post), and after 12 weeks (follow-up) for all participants. Participants in the intervention groups received access to the respective intervention immediately, whereas participants in the CG were given access to one or both interventions of their own choice after 12 weeks, that is, completion of the last assessment. Participating centers of the trial were the outpatient clinics of the Training Center for Psychological Psychotherapy OWL (Bielefeld, Germany), the Centers for Psychotherapy Dortmund and Münster (all training centers of the German Association for Behavior Therapy-Deutsche Gesellschaft für Verhaltenstherapie, DGVT), and the Center for Mental Health and Psychotherapy (Department of Psychology and Psychotherapy, University Witten/Herdecke, Germany).

Eligibility Criteria

Inclusion criteria were (1) current diagnosis of a mental disorder according to the ICD-10 (International Statistical Classification of Diseases, Tenth Revision); (2) recruitment at one of the participating outpatient clinics listed earlier; (3) interest to receive psychotherapy at one of the centers, but not currently enrolled; (4) aged at least 18 years; and (5) reliable internet access. Exclusion criteria were (1) current severe episode of major depression; (2) current psychotic disorder; (3) acute suicidal tendency; (4) other severe mental disorders, for example, current alcohol or substance abuse or current bipolar disorder; and (5) insufficient German language skills. Diagnostic criteria were assessed via a clinical assessment or structured clinical interview. Access to face-to-face psychotherapy after enrollment was not restricted while participants were working with the programs, mirroring naturalistic conditions.

Recruitment, Randomization, and Trial Organization

Participant recruitment and data collection started in April 2020 and ended in July 2022 (thus mostly during COVID-19 lockdowns in Germany). Participants were recruited during registration to regular psychotherapy in the described outpatient centers in Germany. Recruiting outpatient centers received information about the study from a member of the study team [56]. Information on the study and the interventions (REMOTION and Res-Up!) was given to interested individuals coming to the institution for a face-to-face consultation session with additional written information by trained psychotherapists at all centers. Given the long waiting times for psychotherapy in Germany (ranging from 3 mo to 1.5 y), the consultation generally does not mark the beginning of psychotherapy but is scheduled to evaluate whether the person has a clinically relevant psychiatric disorder, and clients are added to waiting lists for psychotherapy afterward. All participants were required to give their informed written consent before enrollment in the study, which was solicited by the respective practitioner and handed to one specified member of the study team, including necessary information (name, birthdate, diagnoses, date of recruitment, and email). After proving study eligibility, participants were sent an email from the study team with an individualized code to the baseline measure via Qualtrics software, using the current version during the period of data collection, 2020-2022 [57,58]. Answering individual questionnaires took approximately 30 minutes at every assessment point. After baseline completion, participants were randomly assigned by a randomized computer generator (in Qualtrics [57]) to 1 of the 3 treatment arms. As completion of the interventions took approximately 6 weeks, posttreatment measures were provided 6 weeks after baseline. Follow-up measures were provided to participants 12 weeks after baseline; follow-up was set this close to the post measure to reach participants with minimal overlap with face-to-face psychotherapy. Participants in the CG were informed that access to the programs would be granted after completing follow-up. In the experimental conditions, participants received an email with their online access to either (1) REMOTION (focusing on emotion regulation) or (2) Res-Up! (focusing on resilience) according to their randomly assigned group. Participants were given the option to contact a member of the study team via text messages during the intervention (“guidance on demand” and “minimal human guidance” [59]) but were not actively contacted by the team (excluding reminders to complete the interventions or questionnaires and a welcome message). Ultimately, 421 participants were randomized into 3 arms with a 1:1:1 allocation ratio (CG: n=141, REMOTION: n=141, and Res-Up!: n=139). Participants in the sample showed a variety of different mental disorders with varying symptom severity, which matches the transdiagnostic approach of the interventions.

Sample Size Calculation

G*Power, current version before the period of data collection 2020 (G*Power is distributed from this web page [60]), [61] was used to calculate the sample size. We planned to detect small effect sizes of Cohen’s d of 0.20 (respectively, Cohen’s f=0.1) regarding the time×group interaction for the 2 active conditions, powered separately, compared to the CG at a Bonferroni-corrected α error level of 0.006. According to the power analysis, a sample size of 103 participants in each of the 3 study arms was required to detect a statistically significant difference with a power (1−β) of .80. Results differ from the study protocol due to the previously uncorrected α level. Due to the high dropout rate (54.3% instead of the estimated 25%), which occurred early in the study during COVID-19 lockdowns, the number of recruited participants was increased during the course of the study to allow additional per-protocol analyses. Dropout rates from internet-based interventions are a frequently reported problem [54,62,63]. We tried to accommodate for this initially by choosing an at least minimally guided format, thus giving participants the possibility to contact the study team via text messages throughout the whole study, which should have increased adherence to the internet-based programs [64].

Interventions

Overview

Participants of both interventions were asked to complete 1 module per week. A variety of psychotherapy elements (eg, cognitive behavioral therapy and positive therapy) were integrated in the program, including video, text, and audio, along with different types of exercises. Participants were reminded to continue working if they had not used the intervention for an entire week but received no other active guidance component (such as weekly emails). However, participants were able to contact the study team via text messages, if any questions occurred (guidance on demand). To increase adherence, reminder messages to complete the modules on time were sent. Completion of both programs was intended to take approximately 6 weeks.

REMOTION as Compensation-Oriented Intervention

REMOTION is a web-based intervention, with password-protected access managed and provided by the University of Bern (Switzerland). A detailed theoretical background for the intervention can be found in the study by Bielinski et al [42]. The goal of the intervention is to improve emotion regulation and reduce patients’ symptom severity. Participants first complete an introductory module and then work on 1 of 5 modules per week; completing a module takes approximately 1 to 2 hours. Figure 1 shows the modules and content of the intervention.

‎

Res-Up! as Capitalization-Oriented Intervention

Res-Up! is a web-based intervention focusing on patients’ strengths and positive experiences, developed and managed at the University Witten/Herdecke (Germany) and provided via Minddistrict [69]. It is based on the PMR [48], which is a positive cognitive intervention that uses patients’ existing strengths to overcome problems. Participants work on 1 of 5 consecutive modules (Figure 2) per week; completing a module takes approximately 1 to 2 hours. A detailed description of Res-Up! can be found in the study protocol [56].

‎

Control Group

Participants in the CG answered the questionnaires before randomization as well as 6 and 12 weeks later. Access to the web-based intervention of their choice was given after completing the 12-week follow-up assessment. During the waiting time, participants were allowed to stay on the wait list or start psychotherapy as they would usually do.

Measures

Overview

Demographic information of participants was recorded at baseline. Information concerning additional psychotherapy received by participants during the course of the study was gathered at baseline, 6 weeks, and 12 weeks. Participants were asked if they were currently attending psychotherapy sessions (“yes” or “no”). Patient’s diagnostic status was obtained by an experienced psychotherapist at the respective outpatient clinics during the initial interview, either by conducting a structured clinical interview I (German version) for diagnostic [72,73] or clinical assessment, depending on the center’s routine practice. Outcomes were gathered online via link to online self-report questionnaires based on Qualtrics software [57]; for a detailed overview, see Table A in Multimedia Appendix 1. Data collection was supported by email reminders. Irrespective of whether the participant started parallel face-to-face treatment or the state of completion of the program, assessments occurred at the given time points. Participants received weekly reminders via email, up to a maximum of 3, if they did not fill out questionnaires.

Primary Outcome Measure

General symptom severity, measured with the Brief Symptom Inventory-short form (BSI-18; German version [73], 2011) of the SCL-90-Revised (SCL-90R [74]), was the primary outcome measure. The BSI-18 consists of 18 items and is a frequently used questionnaire to measure general symptom severity, with 3 subscales: somatization, depression, and anxiety (Cronbach's α=0.85-0.89), with good psychometric properties [75], comparable to those of the SCL-90R [74]. In our sample, Cronbach's α ranged from 0.88 to 0.91.

Secondary Outcome Measures

Emotion regulation was assessed via the following two instruments: (1) self-assessment of emotion regulation skills (SEK-27; Fragebogen zur standardisierten Selbsteinschätzung emotionaler Kompetenzen [76]), a German 27-item self-report measure of emotion regulation skills with high reliability and validity (Cronbach's α=0.90; in our sample=0.93-0.95); and (2) questionnaire assessing the acceptance of unpleasant and pleasant emotions (FrAGe; Fragebogen zur Akzeptanz von Gefühlen [77]), a German 32-item self-report measure of acceptance and suppression of pleasant and unpleasant emotions with good reliability and validity. In our sample, Cronbach's α was 0.93-0.95.

Resilience was assessed via the following two instruments: (1) Witten Strengths and Resource Form (WIRF [78]), a German 37-item self-report of personal and external strengths with high reliability (Cronbach's α=0.84-0.88; in our sample=0.91-0.94); and (2) the Connor-Davidson Resilience Scale (CD-RISC-10 [79]), an internationally used 10-item self-report of individual resilience with high reliability (Cronbach's α=0.81-0.90; in our sample=0.84-0.88).

Severity of depression was assessed with the German version of the Patient Health Questionnaire-9 (PHQ-9 [80]). The PHQ-9 is an internationally used 9-item self-report for screening, diagnosing, monitoring, and measuring the severity of depression with a high retest reliability and validity (Cronbach's α>0.86; in our sample=0.81-0.86).

Self-esteem was assessed with the Rosenberg-Self-Esteem Scale (RSES [81]), an internationally used 10-item self-report of general self-esteem with high reliability and validity (Cronbach's α=0.72-0.85; in our sample=0.91-0.92).

Self-compassion was assessed with the Self-Compassion Scale (SCS-D [82]), an internationally used 26-item self-report of self-compassion with high reliability and validity (Cronbach's α>0.90; in our sample=0.90-0.93).

The therapeutic alliance between participants and the web-based intervention or study team was measured with an adapted version of the Working Alliance Inventory for guided internet interventions (WAI-I [83]) at the end of the intervention (6 wk) and at follow-up (12 wk). The WAI-I is a 12-item self-report scale. For the purpose of this study, the term “psychologist” was substituted with the term “study team” in the questionnaire. The WAI-I has good internal consistency at total and subscale level (Cronbach's α between 0.92 and 0.94; in our sample=0.92-0.94).

Statistical Analyses

Data were analyzed using an intention-to-treat (ITT) approach, including all randomized patients in the outcome analyses (Figure 2), using R Statistical Software (version 4.3.2; R Core Team 2021 [84]) and SPSS (version 29; IBM Corp). For categorical data, counts or percentages were reported. Effect sizes of all between-group and within-group pre-to-follow-up changes were computed as Cohen’s d [85], using an ITT approach unless stated otherwise. In addition, reliable change indices (RCIs [86]) of changes were analyzed by subtracting T1 means from T3 means of every participant, divided by the SE of differences between the 2 scores of the respective (sub)scales [86]. Scores below 1.96 were considered as relevant decreases, whereas scores above 1.96 were considered as relevant increases. Depending on the questionnaire, this indicated improvements or reductions in the respective factors.

Effects of the intervention and CGs on the primary and secondary outcome measures across time were analyzed with multilevel models (MLMs). MLMs are recommended for ITT analyses with missing data because of the possibility to accommodate for larger amounts of missing data without having to exclude or impute data, not depending on limited assumptions about the variance-covariance matrix. Previous research showed that the power of MLM is comparable to (or even higher than) power of ANOVA designs [87-89].

MLMs were used to observe changes in symptom severity (primary); emotion regulation and resilience (secondary); and depressive symptoms, self-esteem, and self-compassion (other outcomes). Three-level models were used, considering the repeated assessments (level 1) as nested within patients (level 2), nested within locations (level 3). The base model 0 was calculated using random intercepts with fixed slopes on the patient level, and no predictors were included. Pseudo-R² was used as a measure of effect size for level 1 [90], which represents the amount of explained level 1 variance, that is, the amount of within-patient fluctuations in the respective main predictor, accounted for by the respective model. The intraclass correlation coefficient was used as a measure of the proportion of variance for the random effect within level 2 and can be interpreted as the expected correlation between 2 randomly drawn level 1 units within a given randomly drawn level 2 unit [91]. The magnitude of this measure can be interpreted similarly to a correlation coefficient, thus representing an effect size index [91,92]. Model fit of competing models was compared by means of likelihood ratio tests for nested models and the Akaike Information Criterion, with smaller values indicating a better fit. Time was coded 0, 6, and 12, assuming a linear trend with realistic intervals according to the measurement points at baseline, 6 weeks, and 12 weeks, and included as the main predictor in model 1. Between-group effects were added in model 2 for the 3 arms (REMOTION, Res-Up!, and CG) to control for variance between the groups before treatment. In the final model 3, we added the interaction time×group as the main predictor to observe possible differences in the change of the respective outcomes between the groups across time. Simple slope analyses were conducted to examine within-group changes over time for each outcome. For additional effect size calculation in the case of MLMs, we used a Cohen’s d equivalent [93], which was calculated by multiplying the difference in the rate of change in the outcome between 2 groups per unit of time (b) by the number of time points minus 6 (as time was coded 0, 6, and 12) (duration) and dividing that by the pooled SD of the 2 respective groups of the means of the raw scores at baseline (either REMOTION and CG or Res-Up! and CG) [94]. As this is a Cohen’s d equivalent, effects are considered small at 0.20, medium at 0.50, and large at 0.80 [85]. A Bonferroni-corrected α level of 0.006 was applied to adjust for multiple (9 measurement instruments overall) comparisons [95]. Results were reported in accordance with CONSORT (Figure 3; Consolidated Standards of Reporting Trials; [96]) and CONSORT-EHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile Health Applications and Online Telehealth) checklists [97].

‎

Ethical Considerations

The study was conducted according to local regulations and the Declaration of Helsinki. The study was approved by the ethics committee of the University Witten/Herdecke (221/2019). Written informed consent regarding the gathered data and following analyses as well as scientific publications in an anonymous form was obtained from all patients by the recruiting therapists in outpatient centers. Neither recruiting therapists nor patients received compensation in any form. Participating patients gained access to the treatments for free (CG after 12-wk follow-up measure completion). The trial was preregistered with ClinicalTrials.gov (NCT04352010). A study protocol was published prior to the study [56]. Data were only analyzed in an anonymous form and deidentified by a member of the research team.

Results

Sample Characteristics

Groups did not differ concerning demographics and diagnosis, comorbidity, and medication at baseline (Table 1). Participants showed a wide variety of mental disorders and comorbidity, with depression being the most common diagnosis (48.46%). Figure 3 displays the flow of patients through the study.

Descriptive statistics for all measures across all time points (ITT) are displayed in Table 2. Effect sizes reported in Table 2 only were calculated for participants with pre- and follow-up data (completer). All further analyses, unless stated otherwise, were done with the ITT sample.

Primary Outcome

In the first set of analyses, we focused on between-group differences in overall symptom reduction. Therefore, we examined whether the 3 groups differed in total scores of the primary outcome measure BSI-18, as well as on its 3 subscales—somatization, anxiety, and depression—within the groups and across time (Table 3).

Results of the MLMs showed that the largest proportion of variation in symptom severity (72.0%) could be attributed to differences between patients, while only 2% could be attributed to differences between locations. In model 1, time was added as a predictor. Symptom severity decreased significantly across the 3 measurements (b=−0.02; P<.001). The model fit improved significantly (χ²₁=36.1; P<.001), resulting in a slightly improved pseudo-R² 0.01 for fixed effects and 0.75 for total effects. For the subscales, the model fit improved significantly for anxiety (χ²₂=29.3; P<.001) and depression (χ²₂=32.6; P<.001), while no significant improvement was found for somatization (χ²₁=7.3; P<.01). In model 2, we added the 3 arms (ie, CG, REMOTION, and Res-Up!) to analyze differences between groups. Model fit did not improve (χ²₂=0.6; P=.74; subscales: somatization: χ²₂=0.7; P=.70; anxiety: χ²₂=0.5; P=.79; and depression: χ²₂=1.2; P=.54), suggesting that symptom severity at baseline did not differ between the 3 conditions. In the final model (model 3), we examined the time×group interaction to examine whether the change in symptom severity differed between the 3 arms over the course of the study. Model fit did not improve significantly for the GSI (χ²₂=1.3; P=.52) or the subscales (all P values >.05), suggesting that the change in symptom severity did not differ between the 3 conditions over time. Consequently, our assumption that there is a stronger symptom reduction in the experimental conditions compared to the CG was not supported by these results.

Simple slope analyses for time effects of symptom severity showed significant decrease over time in all 3 groups (all P<.006), except for BSI-18 somatization, which did not reach significance in any group (all P>.006).

Secondary Outcomes

Emotion Regulation

For the secondary outcome analysis on emotion regulation, we focused on between-group differences in coping with and regulating emotions. Therefore, we examined whether the 3 groups differed in the SEK-27 and the FrAGe scores, with the 2 subscales, acceptance of negative emotions (NE) and acceptance of positive emotions (PE), within the intervention groups and across time.

The results of the MLMs showed that the largest proportion of change in emotion regulation (SEK-27: 69%, FrAGE NE: 63%, and PE: 79%) could be attributed to differences between patients, while no (SEK-27: 0%, FrAGE NE: 0%, and FrAGE PE: 0%) variation in emotion regulation skills could be attributed to differences between locations. In model 1, emotion regulation increased significantly for the SEK-27 (b=0.44; P<.001) and the FrAGe NE (b=0.02; P<.001) but not for the FrAGe PE (b=−0.01; P=.91). Model fit improved significantly for the SEK-27 (χ²₁=28.6; P<.001) and the FrAGe NE (χ²₁=21.8; P<.001), but not for the FrAGe PE (χ²₂=0.01; P=.91). When adding the 3 conditions in model 2 to find differences between groups, model fit did not improve for the SEK-27 (χ²₂=0.1; P=.95), the FrAGe NE (χ²₂=3.3; P=.19), or the FrAGe PE (χ²₂=0.5; P=.77), suggesting that emotion regulation at baseline did not differ between the 3 conditions.

In the final model 3, model fit showed no significant improvement for the SEK-27 (χ²₂=5.0; P=.08). There were no significant group interaction effects for (1) time×REMOTION (b=0.37; P=.06) and (2) time×Res-Up! (b=0.36; P=.07) when compared to time×CG. Regarding FrAGe, model fit did not improve significantly for the subscale FrAGe NE (χ²₂=9.7; P=.008), taking into account the adjusted alpha level of .006. For the FrAGe NE subscale, compared with time×CG, we found highly significant differences between time×REMOTION (b=0.03; P<.001), but not for time×Res-Up! (b=0.02; P=.06) groups. Model fit did not improve significantly for the FrAGe PE (χ²₂=3.2; P=.20). Results for group interactions for the FrAGe PE showed no effect between time×REMOTION (P=.09) or time×Res-Up! (P=.81) compared to time×CG. Plots are depicted in Figure 4. Consequently, our assumption that there is a stronger increase in the experimental conditions is supported for emotion regulation in REMOTION compared to CG with respect to FrAGe NE only.

‎

Simple slope analyses for time effects showed the following results: SEK-27 improved significantly over time in REMOTION (P<.001) and Res-Up! (P<.001), but not in the CG (P=.05). FrAGe NE showed significant increases in REMOTION (P<.001) and Res-Up! (P=.002), while changes in the CG were not significant (P=.30). FrAGe PE did not show significant change in any group (all P>.006).

Resilience

For the secondary outcome analysis concerning resilience, we focused on between-group differences in improvements of resilience. Therefore, we examined whether the 3 groups differed in the CD-RISC-10 and the WIRF scores within the intervention groups and across time.

Results of the MLMs showed that the largest proportion of variation in resilience (CD-RISC-10: 75% and WIRF: 68%) could be attributed to differences between patients, while the lesser to none (CD-RISC-10: 0%; and WIRF: 3%) could be attributed to differences between locations. In model 1, resilience increased significantly for both the CD-RISC-10 (b=0.01; P<.001) and the WIRF (b=0.01; P<.001). The model fit improved significantly for the CD-RISC-10 (χ²₁=17.7; P<.001) and the WIRF (χ²₁=11.9; P<.001). In models 2 and 3, no significant changes or interactions could be observed for the CD-RISC-10 (model 2: P=.44; and model 3: P=.33; interactions compared to time×CG: time×REMOTION, b=0.00; P=.57; and time×Res-Up!, b=0.01; P=.13). Model fit did not improve for the WIRF (model 2: P=.20; and model 3: P=.60; interactions compared to time×CG: time×REMOTION, b=0.00; P=.52; and time×Res-Up!, b=0.01; P=.34). Detailed results can be found in Table B1 in Multimedia Appendix 2.

Simple slope analyses for time effects showed the following results: CD-RISC-10 improved significantly only in the Res-Up! group (P<.001). Changes in REMOTION (P=.04) and the CG (P=.06) were not statistically significant at the corrected level. WIRF improved over time in Res-Up! (P=.02), but this effect did not reach the corrected α threshold. No significant changes were found in REMOTION (P=.05) or the CG (P=.12).

Other Outcomes

Depressive Symptoms, Self-Esteem, and Self-Compassion

For further outcome analyses, we focused on between-group differences in reducing depressive symptoms, improving self-esteem, and increasing self-compassion. Therefore, we examined whether the 3 groups differed in the change of scores of the measures PHQ-9, RSES, and SCS within the intervention groups and across time.

The results of the MLMs showed that the largest proportion of variance (PHQ-9: 70%, RSES: 81%, and SCS: 77%) could be attributed to differences between patients, while little to no variation (PHQ-9: 1%, RSES: 0%, and SCS: 0%) could be attributed to differences between locations. There was an overall time effect for the PHQ-9 (model 1: χ²₁=39.3; P<.001), but no other effects across all models (model 2: P=.86; and model 3: P=.16; interactions compared to time×CG: time×REMOTION, b=−0.10; P=.06; and time×Res-Up!, b=−0.01; P=.84). No effects were found, taking into account the adjusted α level of .006, for the RSES (model 1: χ²₁=5.3; P=.02; model 2: P=.21; and model 3: χ²₂=9.5; P=.009; interactions compared to time×CG: time×REMOTION, b=0.02; P=.009; and time×Res-Up!, b=0.01; P=.01). Overall time effects were found for the SCS (χ²₁=39.7; P<.001), but no further effects or interactions (model 2: P=.28; model 3: P=.10, interactions: all P values >.05) could be found. Detailed results can be found in Table B2 in Multimedia Appendix 3.

Simple slope analyses for time effects showed the following results: PHQ-9 significantly decreased over time in all groups: REMOTION (P<.001), Res-Up! (P=.004), and the CG (P<.001), with all values meeting the Bonferroni-corrected significance threshold. RSES results did not meet the corrected significance threshold for REMOTION (P=.007) and Res-Up! (P=.01), and no significant change was observed in the CG (P=.46). SCS significantly increased across all 3 groups (all P<.006).

Face-to-Face Psychotherapy During the Study

Both REMOTION and CG had a higher rate of parallel regular face-to-face psychotherapy at one (intermittent) or all (2 or 3) of the assessments (Table 4). Answers were categorized as follows: “never,” no concomitant psychotherapy during the whole study; “intermittent,” concomitant psychotherapy at 1 or 2 of 3 time points; and “continuous,” concomitant psychotherapy at all 3 time points.

Adherence to the Web-Based Interventions

REMOTION

Of all patients randomized to the REMOTION group, 18.4% (n=26; Table 5) never started the intervention, despite getting access. On average, participants completed less than half (3 modules) of the program (completed modules, never started=0 modules: mean 2.48, SD 2.12). A module was considered completed if all pages of the module were visited at least once.

Res-Up!

Of all patients randomized to the Res-Up! group, 2.9% (n=4; Table 5) never started the intervention, despite getting access. On average, participants completed less than half (2.5 modules) of the program (completed modules, never started=0 modules: mean 2.32, SD 1.82). A module had to be fully filled out and sent in to be considered completed and to gain access to the consecutive module.

Patients’ Working Alliance

Overall, patients’ ratings of the working alliance stayed the same for REMOTION from T2 to T3 and improved slightly for Res-Up! (Table C in Multimedia Appendix 4). No significant differences between the working alliance ratings in the REMOTION and Res-Up! groups were found in paired t tests (all P values>.05).

Reliable Change Index

To assess clinically significant individual-level changes, RCI percentages were calculated for each outcome, with thresholds set at +1.96 and −1.96. Scores were calculated for participants with pre- and follow-up data (for a detailed description, see Table D in Multimedia Appendix 5). For measures where improvement is indicated by symptom reduction (BSI-18 and subscales and PHQ-9), a higher proportion of participants in the CG exhibited reliable improvement compared to the intervention groups. Specifically, 8.5% of CG participants showed reliable improvement on the PHQ-9, compared to 5.7% for REMOTION and 5.8% for Res-Up! Across all BSI subscales and in all groups, deterioration rates remained low, with REMOTION generally showing the lowest rates of reliable worsening.

For outcomes expected to increase with psychological improvement (emotion regulation, resilience, self-esteem, and self-compassion), Res-Up! showed the highest proportion of reliable improvement in resilience (CD-RISC-10; 7.2%) and emotion regulation (SEK-27; 8.6%). The CG exhibited the strongest positive changes in resource use (WIRF; 14.2%), self-compassion (SCS; 12.1%), and self-esteem (RSES; 12.1%). Reliable change was negligible across groups for the FrAGe PE and FrAGe NE subscales, indicating limited effects on the acceptance of positive and negative emotions. Overall, the CG displayed unexpectedly strong rates of reliable improvement on both symptom- and strength-based measures.

Discussion

Principal Findings

This study compared 2 transdiagnostic web-based self-help intervention programs, focusing on either emotion regulation as a compensation-oriented intervention, REMOTION, or resilience as a capitalization-oriented intervention, Res-Up!, to a CG. Patients were recruited in routine psychotherapeutic outpatient practice during waiting times. All groups improved significantly over time on symptom severity (BSI-18) as the primary outcome as well as on most secondary outcomes: emotion regulation (SEK-27 and FrAGe) and resilience (CD-RISC-10 and WIRF), as well as on depressive symptoms (PHQ-9) and self-compassion (SCS), but not self-esteem (RSES). Contrary to our hypotheses, we could not find additional improvements of symptom severity, resilience, depressive symptoms, self-esteem, or self-compassion in the intervention groups compared to the CG. Solely concerning one emotion regulation measure, the FrAGe NE, a significantly greater increase of the acceptance of negative emotions was found for REMOTION compared to the CG, with a small effect (d=0.21). Further sensitivity analyses of our data are needed to examine additional moderators (eg, face-to-face therapy during the intervention and the number of completed modules) and completer data results.

Implications

The main result of our study is that the active web-based treatments in our study, REMOTION and Res-Up!, overall did not outperform the CG in most outcomes. Only with respect to REMOTION and the acceptance of NE, our findings suggest that there might be a benefit in participating in web-based interventions for emotion regulation while waiting for psychotherapy. This aligns with findings from the pilot trial, where a sensitivity analysis showed the potential beneficial effects of REMOTION, provided as an add-on to psychotherapy, on emotion regulation [43]. It does not align with previous findings for Res-Up! as a stand-alone intervention on a sample recruited from the general population in German-speaking countries [53], as participants showed significant improvements for resilience (CD-RISC-10 and WIRF), emotion regulation (SEK-27), self-compassion (SCS-D), and self-esteem (RSES).

According to recent literature, focusing on internet-based interventions during waiting time for psychotherapy, that include waiting list CGs, this result pattern of equivalent outcomes in designs with comparisons to a CG seems to prevail [13,99,100]. Our findings align with previous less successful research on popular internet-delivered psychotherapy programs (eg, MoodGYM and Deprexis) during waiting times when compared to care-as-usual [99-101] and especially their difficulties considering implementation in routine care (eg, low uptake, high dropout, and low adherence). Even successful RCTs, where symptoms of depression significantly decreased more in the treatment group (eg, GET.ON Mood Enhancer) compared to a waiting list CG, report significant improvements for waiting list CGs [55]. A meta-analysis on the effects of waiting for face-to-face treatment found changes with small effect sizes in the waiting list CGs of over 29 RCTs, indicating positive influence of being on the waiting list for a treatment compared to not [102]. Participants in our CG were informed that they would get access to one or both programs of their choice after completing all 3 assessments. This might have greatly affected our findings, as the information of being on a waiting list alone can influence the reduction of symptoms or improvement of protective factors [103,104]. Therefore, the significant improvement of the CG across all measures and time points might be due to said factors. Furthermore, our findings might be explained by the heterogeneity within the groups with respect to parallel face-to-face psychotherapy, as it may have influenced the overall process, which could explain why incremental effects of the internet-based treatments were not detected [99]. This is especially relevant because the percentages of people beginning face-to-face therapy highly differed between groups. Access to psychotherapy was not limited at any time during the study, as we did not expect participants to receive therapy so quickly after their first visit to the outpatient centers because of the usually long waiting times in Germany (>3 to 6 mo). As there was no a priori control of this factor, the percentage of participants receiving therapy varied greatly across conditions: While only 28.6% of patients in Res-Up! received additional regular psychotherapy at some point or throughout the course of the study participation, 37.1% in REMOTION and 41.4% in the CG did (Table 4).

It has to be taken into account that only half of the eligible patients filled out the baseline questionnaires. This might have been due to different expectations concerning the initial phase of participation. Patients might have not been expecting a questionnaire to be the start of the study participation but a treatment, although they were informed about this by mentioned trained psychotherapists. This might also explain why patients dropped out later, for example, after getting access to a program when they wanted to do the other, or because the transdiagnostic approach did not meet their expectation of a disorder-specific or disorder-focused treatment approach. Recent studies emphasize the importance of individual tailoring of internet-based treatments to increase effectiveness and adherence [105], while others, especially looking at transdiagnostic approaches for anxiety and depression, claim that universal transdiagnostic approaches seem to be superior to tailored ones [19]. Further research is needed to evaluate whether universal or tailored transdiagnostic approaches are more beneficial for treatment outcome as well as adherence and whether that might have been part of the reason for the high dropout rates in this study.

Concerning the question of whether a compensation-focused (REMOTION) or a capitalization-focused (Res-Up!) approach might be more appropriate in an online setting, our findings do not allow an answer, as there were no significant outcome differences between the 2 interventions. In future research, a more comparative approach would be needed in addition to more distinct interventions for compensation and capitalization.

To determine whether web-based interventions during waiting times in routine outpatient care could be effective, more studies are needed, and parallel psychotherapy should be analyzed more closely and in further RCTs.

Limitations

Several limitations, first in study design and second in intervention design have to be considered. The possibility to generalize our findings to the general population of outpatient psychotherapy patients in Germany is limited because we applied exclusion and inclusion criteria and recruited a selected sample. The trial showed high dropout rates for measurements of 54.3% overall and especially for the treatment groups 64.3% (REMOTION: 67.1%, Res-Up!: 61.4%, and CG: 34.3%), which we tried to accommodate for by pursuing an intention-to-treat-analysis via MLMs and a higher number of participants recruited. Moreover, only 12.8% in REMOTION and 23.7% in Res-Up! completed all modules of the respective intervention, but intervention dropouts were not excluded due to the ITT approach. Additionally, effects of the treatments might have been attenuated further because participants, on average, only completed half of the modules of respective programs (Table 5), which is known to be a substantial influence on the effectiveness [99,100,106]. It is unclear whether patients with less than two-thirds or even half of the treatments completed even received an adequate dosage of the respective treatment’s interventions [100]. While high dropout rates are a frequently reported problem of internet-based interventions [13,54,62], they are likely to be one of the main explanations for our results, for measurement dropouts and especially for the intervention dropouts. Additionally, unguided internet-based self-help interventions without any contact between patient and provider are less effective than therapist-supported internet interventions [64]. As we offered only minimal support and participants had to come forward on their own to receive it, this structure of the internet-based interventions may have impacted the outcomes [59,107]. It may also be possible that we missed occurring longer-term improvements because of the relatively short follow-up period of 12 weeks. Although it might also have led to an even higher dropout rate.

As we did not include sensitivity analyses, in accordance with the ITT approach in the study protocol [56], it limits possible conclusions to our ITT sample and a priori determined outcomes. Additional sensitivity analyses including completer analyses, for measurement points as well as completed modules, or considering parallel access to face-to-face therapy might furthermore have improved the robustness of our analyses and improved the generalizability of our results.

Future Directions

Results from this study highlight difficulties for implementation and limits of internet-based effectiveness of such interventions during waiting times for psychotherapy. In particular, the study created new knowledge on web-based interventions focusing on resilience and emotion regulation, aligned with capitalization and compensation approaches to psychotherapy. Further analyses may want to examine the interaction of the intervention foci with patient variables: If patients with a particularly high level of adaptive emotion regulation strategies especially benefit from an intervention focusing on the improvement of emotion regulation strategies, it would imply a capitalization change process. If patients with a particularly low level of adaptive emotion regulation strategies benefit particularly from an intervention focusing on the improvement of emotion regulation strategies, this points to a compensation change process. Respective interaction effects between patients’ resilience and a resilience-focused intervention can be hypothesized. Thus, in further analyses, differential effects of patients’ initial levels of resilience and emotion regulation competencies on the outcome of a resilience and emotion regulation intervention should also be explored. Potential limits of internet-based interventions during waiting times are especially present when waiting times are shorter, and there is an overlap of intervention and face-to-face psychotherapy. Therefore, better ways to implement internet-based interventions in the routine setting are direly needed. The results of our study implicate that a focus on more integrated ways of combining such interventions with face-to-face psychotherapy is needed rather than implementing them as stand-alone during patient wait times.

Conclusions

Therapy Online Plus is a multicenter RCT, comparing a care-as-usual–only CG with two care-as-usual plus 1 of 2 transdiagnostic web-based interventions. One intervention directed at compensating emotion regulation problems—REMOTION—and the other directed at capitalizing on patients’ strengths—Res-Up!—in a clinical sample. Results showed significant improvements across time but no additional benefits of the intervention groups on general symptom severity compared to the CG. Slight benefits of REMOTION on emotion regulation in the form of a higher acceptance of NE compared to CG could be found.

The study extends knowledge on potentials and challenges of implementing transdiagnostic internet-based interventions in routine outpatient psychotherapy settings during waiting times and beyond. Given the time effects found for most outcomes, we can safely assume that both treatments had a positive effect on participants, but no incremental effects when compared to the CG. This has been found in many RCTs for internet-based interventions. Future research should focus on better ways to implement internet-based interventions in routine outpatient care, possible long-term outcomes, and blended care approaches. Furthermore, they will have to evaluate the economic effectiveness and efficiency of transdiagnostic internet-based interventions during waiting times in routine outpatient care. From our findings and a high number of other studies in the field, we have to assume that internet-based interventions, especially transdiagnostic ones, do not seem to be more effective than simply waiting for psychotherapy.