Internalizing symptoms are among the leading causes of disability in the United States and worldwide [1]. For example, major depression affects at least 20% of adults in the United States [2]. Although between individuals, depression is variable in its symptom presentation [3,4] and overall prognosis [5,6], at a societal level, it is a leading contributor to the global burden of disability [1]. Between individuals, depression strongly covaries with other internalizing symptoms (eg, generalized anxiety and emotional liability) as part of what has been called an emotional dysfunction superspectrum of symptoms that may reflect difficulties regulating negative emotions [7]. Depression is a major contributor to the emotional dysfunction superspectrum but other internalizing symptoms, such as anxiety, are similarly heterogeneous in presentation and prognosis [7].

Psychological interventions, the most widely studied of which are cognitive behavioral therapies (CBTs), target depression and other internalizing symptoms, partly by improving an individual’s capacity to regulate negative emotions, including increasing the use of cognitive reappraisal [8,9] and decreasing maladaptive avoidance strategies, such as suppressing the expression of emotions. These findings align with a meta-analysis of emotion regulation experiments [10], which reported that regulating emotions through cognitive reappraisal is associated with improvements in emotions (standardized mean difference [SMD] 0.45, 95% CI 0.35-0.56), and expressing emotions (ie, not suppressing their expression) is also associated with emotional outcomes (SMD 0.10, 95% CI 0.01-0.18).

It is widely recognized that current treatment efforts, which are mostly centered around individual interventions delivered by a professional, such as antidepressant medications and face-to-face psychological interventions such as CBTs, have not made a dent in the public health burden of depression and are unlikely to make a substantial public health contribution in the near future [11]. Prior studies have highlighted at least 2 strategies with the potential to reduce the public health burden of untreated depression. One strategy is to identify individuals who are at a relatively high risk of depression onset or relapse [12] or who experience health disparities. For example, cisgender women; lesbian, gay, bisexual, trans, queer, or other sexual minority (LGBTQ+) individuals; people with a prior history of depression; and individuals who are not in education, employment, or training are at high risk for depression onset and relapse after symptom improvement [12]. Individuals with identities that are minoritized based on race or ethnicity or who live in areas with limited access to health providers tend to have lower access to mental health services than their race or ethnic majority counterparts and individuals who live in areas with a high number of health providers [13].

Another strategy to decrease the public health burden of depression is to develop and disseminate interventions that are more scalable than antidepressants and individual face-to-face psychological interventions [11]. Digital mental health interventions (DMHIs) have emerged as some of the most popular treatment modalities because of the ubiquity of internet access. A substantial body of evidence suggests that DMHIs are effective when compared with waiting list controls (WLCs), care as usual, and other controls such as sham apps [11,14-16]. Digital single-session interventions (SSIs) have emerged as promising scalable treatments for internalizing symptoms in youth because SSIs are not time-consuming and can be made freely available [17,18]. In a digital SSI, an individual completes a web-based intervention in 1 sitting. Free-standing brief treatments, such as unguided digital SSIs, are also important interventions for the treatment of internalizing symptoms because engagement with face-to-face therapies and DMHIs in real-world settings is usual brief [17,18]. Meta-analyses in youth suggest that SSIs are more effective than control conditions, such as waiting lists, care as usual, or “placebo” SSIs [17,18].

In the treatment of adult psychopathology, individual face-to-face SSIs have a long history as treatments for specific phobias and obsessive-compulsive disorder [18], and other clinical problems are amenable to highly symptom-focused treatments. Unguided digital SSIs for the treatment of internalizing symptoms in adults have yielded mixed findings. For example, an instructor-led SSI for chronic pain appeared to be more effective than health psychoeducation and noninferior to an 8-week course of CBT [19]. However, in a meta-analysis by Schleider and Weisz [17], the effect sizes were very small for depression as a treatment target or secondary outcomes. Mullarkey et al [20] found no support for the efficacy of an unguided SSI focused on perceived control over anxiety versus a placebo control in a nationally representative sample of 500 adults. Thus, the efficacy of unguided digital SSIs for adult internalizing symptoms is still a matter to be established.

Wasil et al [21] developed an SSI called the Common Elements Toolbox (COMET) that captures elements common to CBT and positive psychology, including behavioral activation, cognitive restructuring, self-compassion, and gratitude. Adding positive psychology interventions to traditional CBT strategies may lead to especially positive outcomes because evidence suggests that interventions that target positive emotions may specifically target positive affect [22,23] or help individuals not helped by traditional CBT [24]. In an initial investigation, undergoing COMET was associated with improvements in perceived coping abilities, and the intervention was feasible and acceptable to graduate students [21]. In the first randomized controlled trial (RCT) evaluating the efficacy of COMET, Steinberg et al [25] reported that COMET reduced depression in undergraduate students when compared with a study skills control condition at 4-week (SMD 0.20) and 12-week follow-ups (SMD 0.17). Nonetheless, these small effects may be promising for scalable treatments, such as unguided digital SSIs.

Given the high potential for dissemination of a free SSI such as COMET and the preliminary evidence for its efficacy, we studied the efficacy of COMET in a population at high risk for depression: paid web-based participants. Researchers are the most familiar with paid web-based participants because they often use them in survey studies and web-based experiments [26]. Existing data suggest that paid web-based participants report very high levels of depression when compared with the general population [27], although somewhat lower than other convenience samples, such as college students [28]. Ophir et al [27] showed that only some of this elevated depression is attributable to inattentive or potentially fraudulent respondents (eg, bots). Their results suggest that sociodemographic characteristics, especially the level of physical activity and sedentary behavior, explain some of the risk for depression associated with web-based work, but most of the variance in the risk for depression in paid web-based participants versus the general population remains unexplained. McCredie and Morey [29] characterized paid web-based participants as being more neurotic and less socially engaged than the rest of the population. This suggests that paid web-based participants are at an elevated risk for depression and have documented risk factors for psychopathology that may ameliorate using psychological interventions such as COMET.

Prior studies suggest that intervention trials with paid web-based participants are feasible. For example, in a factorial RCT exploring variations of cognitive bias modification, Steinman et al [30] recruited 1120 individuals with moderate to severe anxiety and treated them with variations of cognitive bias modification. The results suggested that some variations of cognitive bias modification appeared slightly better at manipulating anxiety-related cognitive biases, but the study did not include a postbaseline assessment exploring the intervention’s effect on anxiety or depression (ie, the study focused on the engagement of cognitive biases with treatment). Mullarkey et al [20] attributed their failure to find statistically significant intervention effects on the characteristics of their sample (ie, recruiting healthy adults vs clinical high-risk groups), their outcome choices (ie, anxiety and perceived control), and the use of active control, which may have limited the finding of small effects.

We conducted an RCT to evaluate the short-term (2 weeks) and long-term (8 weeks) efficacy of the COMET-SSI in paid web-based participants, addressing the limitations of prior studies on SSIs in adult paid web-based participants [20]. We focused on adults with ongoing or previous mental health problems who may be at risk for current symptoms; assessed a wide variety of outcomes, including internalizing symptoms, emotion regulation, and functioning; and used a WLC. We chose to study COMET-SSI given its feasibility and preliminary evidence for its efficacy. We hypothesized that the COMET would lead to small improvements in depression, anxiety, well-being, functioning, and emotion regulation.

This was an RCT that has a preregistration for its design (ClinicalTrials.gov NCT05379881) and analysis plan [31]. It was a 2-arm parallel RCT with a 1:1 allocation ratio, comparing short-term (ie, 2 weeks) and long-term (ie, 8 weeks) depression and anxiety changes in paid web-based participants randomized to receive the intervention COMET or a WLC. This trial aimed to assess the efficacy of COMET on primary and secondary outcomes. We determined the choice of control condition using the Pragmatic Model for Comparator Selection in Health-Related Behavioral Trials [32,33]. This framework specifies that no one control group is optimal, but the primary aim of the trial and limitations and barriers must be considered when choosing a control condition. Although some control groups, such as placebo controls, tend to be more methodologically rigorous than other controls, they may not be the best control group in any specific study. One consideration in designing our trial was that the efficacy of SSIs, in general, has been well-established in youth [17]. Although meta-analyses suggest that the effects of psychological interventions are stronger in adults than in youth [15], the efficacy of SSIs in adults and of the COMET specifically is not as well-established as the efficacy of SSIs in youth. Thus, we chose to compare the COMET with a WLC.

Paid web-based participants living in the United States were sampled from Prolific [34], a web-based platform hosting research studies. The only inclusion criterion we used for the trial was an affirmative answer (ie, responding “yes”) to the question “Do you have—or have you had—a diagnosed, ongoing mental health/illness/condition?” On the basis of this inclusion criterion and sampling from the United States, 14,628 individuals were eligible for the study. Participants were informed that this was an Indiana University study that explored the efficacy of the COMET. No explicit exclusion criteria were applied. However, it could be argued that relatively steady internet access and internet literacy were entry criteria because the study required individuals to log in to Prolific 1 to 4 times.

This study (#12290) was approved by the human subject research ethics review board of Indiana University. The study was described to participants as “testing the efficacy of an online wellness activity called the Common Elements Toolbox (COMET) [which] helps you learn skills that may help to improve your mood and well-being.” Participants were paid a maximum of US $12.02 for participation in the study, a rate of roughly US $8 an hour. This was divided into payments of US $8 for the baseline assessment, irrespective of treatment condition, and 3 payments of US $1.34 for the 2-, 4-, and 8-week surveys. The participants were assured that their study data were deidentified.

The study was hosted in Qualtrics (Qualtrics International Inc), and randomization was performed within the Qualtrics platform, which uses simple randomization. Thus, the treatment allocation schedule was concealed from the investigator and the participants. A study coordinator coded the treatment condition, concealing the coding scheme from the principal investigator (PI). To prepare all analyses associated with this study, the PI used the R (R Foundation for Statistical Computing) [35] programming language to create a synthetic treatment variable that had no correspondence to the actual treatment assignment. Owing to the nature of the intervention, the actual treatment assignment could not be concealed from participants, who were the ones who assessed outcomes.

An a priori analysis in G*Power (version 3.1; Heinrich-Heine-Universität Düsseldorf) [36] suggested that to have 80% power to detect a small treatment effect (SMD 0.20) using a P value of <.05 as the cutoff and with 1:1 allocation, we would need to recruit 788 individuals. We did not factor in attrition because we planned to impute data but aimed to recruit 800 individuals because we had available funding for slightly more than 788 individuals. We posted the study to Prolific on July 5, 2022, and enrolled 800 individuals on the same day. We accepted and compensated participants for the study if they completed the baseline battery of assessments, which would take approximately 15 minutes. As Prolific only counts participants as completing a study if they scroll all the way through to the end, we recruited 861 individuals. Of these 861 individuals, 33 (3.8%) did not complete the baseline survey; therefore, only 828 (96.1%) participants were randomized.

All measures were self-reported web-based assessments administered at baseline (ie, July 5, 2022), two weeks after baseline (July 19 to 21, 2022), four weeks after baseline (August 2 to 4, 2022), and 8 weeks after baseline (August 30 to September 3, 2022). One deviation from the trial protocol is that a day before the end of the 8-week assessment, we reminded participants in either treatment condition who had not yet completed the intervention that the study assessments were available in Prolific.

At the request of a reviewer, we have included additional data attesting to treatment satisfaction with the COMET. At the end of the intervention, participants (n=381) were queried about their satisfaction with the intervention with questions about the program in terms of (1) their approval of it, (2) the degree to which they found it appealing, (3) their satisfaction with it, and (4) their willingness to integrate it into their lives. Responses were provided on a 4-point scale, ranging from 0 to 4. We calculated the mean and SD of the 4 questions. Responses to these questions appeared to be internally consistent indicators of satisfaction with the intervention (ω=0.96, 95% CI 0.94-0.97).

Prior studies suggest that inattentive or potentially fraudulent respondents may provide biased data. Following best practices in the detection of inattentive or fraudulent respondents, we used a multimethod approach to identify inattentive or potentially fraudulent respondents. First, we obtained an estimate of the time it took the participants to complete the baseline portion of the study. Given that the treatment condition would be a significant predictor of the time spent on the study at baseline, we z-scored the duration for each participant within each treatment condition. Respondents taking <2.5 SDs to complete the first part of the study were deemed as completing the study implausibly fast. In addition, at baseline, we administered the 10-item Personality Inventory [46], a measure of the Big Five personality traits. As the 10-item Personality Inventory contains 5 items that are scored in one direction (per subscale) and 5 items that are calculated in another direction, we were able to capture response inconsistency on this scale. We considered respondents as “inconsistent” if >40% of their responses were in an inconsistent direction. We also used RelevantID as a third attention check and the reCaptcha score as a fourth attention check [47]. RelevantID is a proprietary algorithm that analyzes the user’s browser, operating system, and location to provide a fraud score ranging from 0 to 100, with scores >30 usually understood as bots. The reCaptcha scores range from 0 to 1, with a higher score indicating a higher likelihood that the respondent is human. Scores of ≥0.5 are generally considered human responders.

A total of 861 individuals began the study, but 33 (3.8%) did not complete the baseline screening (Figure 1). Most participants met at least one cutoff of clinical significance in the PHQ-9, GAD-7, WSAS, or WHO-5 (732/828, 88.4%), with a median of 2 cutoff values being met. Of all individuals who completed the baseline screening (n=828), a total of 479 (57.6%) met the PHQ-9 screening for major depression and 348 (42%) met the cutoff score for generalized anxiety on the GAD-7. In total, 616 (74.4%) individuals met the WHO-5 screening criteria for major depression. More than half (n=421, 50.9%) of the participants met the WSAS criteria for poor functioning, and 83.3% (n=690) had a lifetime history of antidepressant use. On average, the participants were in their mid-30s, although they reported an early age of onset of symptoms of depression, anxiety, or stress. The female-to-male sex assigned at birth ratio was roughly 2:1. Approximately one-third (n=280, 33.8%) of the participants identified as LGBTQ+. The other demographics are presented in Table 1. No harms were reported by participants.

There were statistically significant differences (χ²₁=24.3, P<.001) between the treatment conditions in the rate of completing the allocated intervention. Individuals in the WLC were more likely to complete the baseline rating task (417/419, 99.5%) than individuals in the COMET were to complete the COMET-SSI (379/409, 92.7%); however, these differences were small. There were no statistically significant differences between the treatment conditions in the rate of return for the 2-week assessment from July 19 to 21, 2022 (χ²₁=0, P=.99); 4-week assessment from August 2 to 4, 2022 (χ²₁=1.1, P=.29); or 8-week assessment from August 30 to September 3, 2022 (χ²₁=0.2, P=.66).

A few individuals were flagged with 1 inattention check (n=40), and only 1 person was flagged with 2 inattention checks. The differences in the rate of inattentive respondents between the treatment conditions were not statistically significant (P=.07). At the request of a reviewer, we reviewed all text responses to the COMET and coded and rated whether they appeared to be valid responses to the prompts (eg, if individuals responded with activities to the behavioral activation module). Of the individuals who reached the first activity (n=408), behavioral activation or “pleasant activities,” all provided answers that were deemed valid responses. Of the individuals who reached the point of cognitive restructuring or “flexible thinking” (n=388), all but 3 (99.2%) individuals provided valid answers. Similarly, of the 388 individuals who reached the “gratitude” component, all but 3 (99.2%) provided valid answers. Of the 381 individuals who reached the final module, self-compassion, 379 (99.5%) provided valid answers. Thus, most individuals appropriately engaged with the web-based content.

Overall, there were no statistically significant differences between the COMET (n=409) and WLC (n=419) in the short-term primary outcomes of depression and anxiety (Table 2). Not only were outcomes not significantly different between the groups but also the magnitude of the differences was quite small (Figure 2A). Similarly, there were no statistically significant differences between treatments in terms of well-being, psychosocial functioning, or emotion regulation. The 8-week data showed an almost identical pattern. In other words, there were no statistically significant differences between the treatment conditions.

Overall, there were no statistically significant differences between the COMET intervention (n=409) and WLC (n=419) in the short-term primary outcomes of depression and anxiety (Table 3). Not only were outcomes not significantly different between the groups but also the magnitude of the differences was quite small (Figure 2B). Similarly, there were no statistically or clinically significant differences between treatments in well-being, psychosocial functioning, or emotion regulation outcomes. The 8-week data showed an almost identical pattern of results, that is, there were no statistically significant differences between the treatment conditions. Multimedia Appendix 1 provides raw scores by treatment over time.

The results with the “per-protocol” sample mirrored those of the imputed and imputed data sets, suggesting no statistically significant differences (WLC: n=417; COMET: n=379; Table 4). At week 2, the differences between the groups were small for depression (SMD −0.06, 95% CI −0.20 to 0.07), anxiety (SMD −0.05, 95% CI −0.18 to 0.09), well-being (SMD 0.02, 95% CI −0.12 to 0.16), functioning (SMD −0.04, 95% CI −0.18 to 0.09), cognitive reappraisal (SMD 0.03, 95% CI −0.11 to 0.16), or expressive suppression (SMD −0.08, 95% CI −0.22 to 0.06),

At week 8, the differences between the groups were small for depression (SMD −0.03, 95% CI −0.17 to 0.10), anxiety (SMD −0.01, 95% CI −0.15 to 0.12), well-being (SMD 0.06, 95% CI −0.08 to 0.19), functioning (SMD −0.02, 95% CI −0.16 to 0.12), cognitive reappraisal (SMD 0.04, 95% CI −0.10 to 0.18), and expressive suppression (SMD −0.07, 95% CI −0.21 to 0.06).

To control for the possibility that our results were accounted for by inattentive respondents, we removed all participants who had failed even a single attention check, leaving us with a sample of 787 individuals (WLC: n=404; COMET-SSI: n=383). Across the 3 time points (weeks 2, 4, and 8), six outcomes (ie, depression, anxiety, well-being, functioning, cognitive reappraisal, and expressive suppression), and 3 data sets (ie, imputed, unimputed, and per-protocol), removing inattentive participants did not change the pattern of results (Multimedia Appendix 1). The largest intervention effect for depression was at week 4 in the per-protocol sample, representing a very small effect (B=−0.63, 95% CI −1.26, to −0.01; P=.046). Nonetheless, this was a very small effect, not corrected for multiple comparisons, which was inconsistent across the data sets and time points. Similarly, there was some evidence of a reduction in expressive suppression in COMET relative to WLC at week 2 (B=0.11, 95% CI −0.02 to 0.24; P=.09), but this was a small effect (SMD 0.07, 95% CI −0.07 to 0.21).

In addition, given that we did not use severity on clinical screening scales as an entry criterion, it is possible that intervention effects may be observed in individuals with more severe psychopathologies. To address this possibility but balance multiple comparisons, we reran analyses for each of the subscales that can serve as screening scales for depression or anxiety (ie, the PHQ-9, GAD-7, WSAS, and WHO-5). For example, we reran the PHQ-9 analyses described earlier in the imputed, unimputed, and per-protocol samples in individuals with PHQ-9 scores ≥10. We reran all the GAD-7 analyses described earlier in the imputed, unimputed, and per-protocol samples in individuals with GAD-7 scores ≥10. The sample size for these sensitivity analyses ranged from 333 to 616. Assuming a P value of .05 (ie, not adjusting for multiple comparisons), we could detect effect sizes ranging from 0.22 to 0.31. Figure 3 shows a summary of effect sizes from the different measures (ie, PHQ-9, GAD-7, WSAS, and WHO-5) at different time points (ie, weeks 2 and 8) for the different data sets (ie, imputed, unimputed, and per-protocol). As shown in Figure 3, these effects were not statistically significant.

On average, participants reported high levels of satisfaction with COMET, suggesting they “agree” to “completely agree” that they are satisfied with the program (mean 3.11, SD 0.79).

We conducted an RCT to evaluate the efficacy of the COMET-SSI versus a WLC in Prolific participants. Our study was powered to detect small differences between the treatment conditions. The sample showed elevated levels of depression and anxiety, with most individuals being flagged on a screening scale for depression, anxiety, or poor functioning. However, our results did not suggest statistically significant differences between the COMET and WLC in any of the psychologically relevant outcomes we assessed, despite most individuals completing the intervention and even when focusing on individuals with more severe symptoms. Notably, participants in the COMET-SSI group were highly satisfied with the intervention, although they did not experience greater symptom improvement than participants in the WLC group. The correlation between participant satisfaction and treatment outcomes was known to be rather small [59].

It has been estimated that in the treatment of depression, effect sizes of approximately 0.24 may be deemed clinically significant [60]. Given that our study was powered to detect smaller differences, we must conclude that if the COMET-SSI has effects in paid web-based participants, the effects are so small that they are clinically questionable. Indeed, the 95% CIs for all of our effect size estimates mostly contained very small values (ie, +0.20 to −0.20). These results are consistent with the findings of Mullarkey et al [20] who did not find a statistically significant effect of an SSI on adult anxiety. Thus, our 2 studies suggest that paid web-based participants may be more treatment resistant than youth and college students. These results are in contrast with the results supporting the efficacy of DMHIs [11,14-16].

Before interpreting our results, it is worth considering the aspects of our study design that may impose limitations on the inferences from these data. First, we recruited paid web-based participants for this study. Paid web-based participants have been noted as a population at high risk for depression [27]. We recruited participants if they had an “ongoing/diagnosed” mental health condition. Although the scores on the clinical screening scales suggest that most people would have met the criteria for a depressive or anxiety disorder if we had screened them (eg, with the PHQ-9), it is possible that we recruited many individuals who would not have benefited from treatment for depression or anxiety and may have benefited from other interventions (eg, substance use treatment and insomnia treatment). We did not recruit participants seeking help in routine care and instead provided treatment to individuals with mental health problems. However, Zhao et al [61] recently reported that psychological interventions can be effective, even when individuals are not seeking treatment. In addition, although Prolific is renowned for its data quality [62] and we reviewed participants’ responses for data quality, we have no way of confirming that respondents honestly engaged with the material after the intervention was completed. Although the COMET intervention asked participants to engage with the material by planning to use CBT and positive psychology principles, as in previous SSI studies, we did not measure adherence to the intervention outside its context of the intervention itself (eg, daily life). Our sample had a relatively high number of individuals who were identified as LGBTQ+. Other studies of unguided digital SSIs have reported high numbers of LGBTQ+ individuals [63,64], suggesting that in addition to a heightened risk of internalizing symptoms, LGBTQ+ individuals may have a greater willingness to engage with these treatments. Future studies should explore whether sexual orientation moderates the treatment outcomes and engagement. Finally, the typical limitations of web-based trials apply to this study, including the lack of concealment of intervention conditions to the participant, as well as the reliance on self-report measures.

This study had several strengths. First, we conducted a rather large study, which was well powered to detect small differences between treatments. This study is the largest assessment of the COMET-SSI and a very large trial as far as psychological interventions go [65]. Second, we preregistered our analytic plan and conducted the trial by concealing the intervention from the investigator and the analyst. Although we measured symptom outcomes of depression and anxiety, dubbed as the “vital signs” of mental health [66], we also assessed 2 important elements of quality of life—well-being and functioning—and 2 emotion regulation strategies—cognitive reappraisal and expressive suppression. In addition, consistent with calls for more generalizable depression research, we used relatively lax study entry criteria [67-69]. Finally, we used a multimethod approach to identify inattentive respondents, and our results suggested that most individuals were attentive.

Although SSIs have shown great promise in the treatment of adolescent internalizing distress [17] and in-person exposure-based SSIs have been very effective in the treatment of fear-based disorders in adults [70], less work exists on unguided SSIs in the treatment of adult internalizing distress. Notably, internalizing distress symptoms such as depression and generalized anxiety have been clinical problems for which it has been difficult to identify interventions that are more effective than controls [71,72]. Future work should explore the efficacy of SSIs for internalizing distress in other adult populations. Dobias et al [63] recently reported that an SSI focused on targeting self-hatred to reduce self-injurious behaviors was not significantly more effective than a control condition SSI in the study’s preregistered outcomes. Mullarkey et al [20] also reported that targeting perceived control over anxiety did not lead to improvements relative to the control condition. These findings, and ours, may suggest that there needs to be more care in the design of trials of SSIs in relation to the populations and clinical problems targeted. For some clinical problems and some populations, SSIs may be effective irrespective of the active mechanisms the interventions target. Alternatively, data-driven approaches may help to identify subgroups of individuals who preferentially respond to specific SSIs [5,6,73,74].