Detailed methods are described in the study by Cheung et al [22]. The study was approved by the institutional review board and preregistered through ClinicalTrials.gov (trial number: NCT02861755). All participants were recruited on the web through platforms such as ResearchMatch, Craigslist, and Reddit. Eligibility was determined using a web-based screener. To be eligible for the study, participants had to (1) have at least mild levels of depression, as indicated by a Patient Health Questionnaire-8 (PHQ-8) depression score of ≥5 [38]; (2) be aged ≥18 years; (3) have daily access to the internet; (4) own a mobile phone; (5) live in the United States; and (6) be able to read and write in English.

Of the 1805 respondents who completed the screener, 1037 (57.45%) were deemed eligible and entered the run-in period. Of these, 58.05% (602/1037) completed at least 4 of 7 days of the emotion reports for the run-in and were sequentially assigned to one of the nine study conditions. Figure 1 shows the CONSORT (Consolidated Standards of Reporting Trials) diagram. There were 485 eligibility forms that were deemed spam. Most of these cases were identified because they enrolled in quick succession on the same day and had suspicious repeating patterns of first or last names. We sent additional screener questions to these cases and none completed them and therefore did not proceed to enrollment.

Across the nine conditions, there were no differences in baseline demographics, depression, or comfort with technology (Table 1). The average age of the participants was 38 years (range 18-80 years). Three-fourths of the sample were female (422/573, 73.6%), three-fourths were White (421/573, 73.4%), and 43% (244/568) were college graduates. Compared with those who did not complete at least 4 of the 7 days of the run-in period, participants who completed the run-in were older (mean 37.95, SD 13.68 vs mean 36.03, SD 12.77; P=.04), more likely to be female (467/633, 73.7% vs 210/323, 65%; P=.008), and had lower baseline depressive symptom severity (PHQ-8: mean 13.85, SD 5.18 vs mean 15.34, SD 5.11); P<.001; Table S1 in Multimedia Appendix 1).

Participants accessed, on average, 5.61 (SD 2.76) of 8 skills. The total number of enhancements that participants received (0-3) did not predict the number of skills accessed (1 enhancement vs 0: P=.63; 2 enhancements vs 0: P=.25; 3 enhancements vs 0: P=.19). We conducted an analysis predicting the number of skills accessed as a function of whether the participant received each enhancement and their two- and three-way interactions. There were no significant main effects for the FC (P=.24) or ODB (P=.36) enhancements. There was a significant main effect of receiving the VB enhancement, such that participants in the conditions that received VB (VB, VB+FC, VB+ODB, and VB+FC+ODB) had a greater number of skills (mean 5.90, 95% CI 5.52-6.31) relative to participants in groups that did not receive the VB enhancement (intervention only, FC, ODB, and FC+ODB; mean 5.30; range 4.96-5.67; χ²₁=5.0; P=.03). This main effect, however, appeared to be driven by a significant VB by FC two-way interaction predicting the number of skills accessed (χ²₁=5.5; P=.02) such that participants in arms that received the combination of both the VB and FC enhancements together (VB+FC and VB+FC+ODB) accessed a greater number of skills (mean 6.42; range 5.87-7.03) than participants who received either enhancement without the other (VB without FC: mean 5.42; range 4.92-5.98; P=.01; FC without VB: mean 5.15; range 4.67-5.68; P=.001). Furthermore, participants who received either VB or FC enhancement without the other did not differ in the number of skills accessed compared with participants who received neither enhancement (neither VB or FC: mean 5.45; range 4.98-5.98; P=.40 to P=.93). No other two-way or three-way interactions emerged with statistical significance (FC by ODB interaction: P=.17; ODB by VB interaction: P=.44; and FC by ODB by VB interaction: P=.94). Taken together, these findings suggest that receiving the VB and FC enhancements in combination increased the number of skills accessed, but receiving either the VB or FC enhancement without the other did not (Figure 2).

The patterns of results were the same for other indicators of adherence (eg, pages viewed or home practice completed) and retention in the study. The details of these analyses are provided in Multimedia Appendix 1.

We examined age, gender, race, ethnicity, comfort with technology, and baseline depression as predictors of adherence and as moderators of the effects of each individual enhancement on adherence. As seen in Table 2, none of these baseline characteristics significantly predicted participants’ adherence to the intervention content (P=.19 to P=.87).

Next, we explored whether the enhancements had a greater effect on adherence depending on demographics, comfort with technology, or baseline depression. There was a significant baseline depressive symptom severity by VB two-way interaction in predicting the number of skills accessed, χ²₃=8.6 (P=.04). Specifically, among participants with moderately severe depression (scores between 15 and 19 on the PHQ-8), receiving VB enhancement was associated with better adherence compared with those who did not receiving VB enhancement (B=1.09; P=.01). Two trends in the data that did not reach conventional levels of statistical significance are worth noting: for those in the mild (5-9) and moderate (10-14) range in the PHQ, there was also a positive association between the VB group and adherence (mild: B=1.00; P=.06; moderate: B=0.80; P=.10). At the highest level of depressive symptoms (scores >20); however, the association was in the opposite direction, with those who received the VB enhancement showing lower adherence compared with those who did not receive the VB enhancement (B=−1.21; P=.08). Given that the effects among participants in 3 of the 4 depression groups (mild, moderate, and severe but not moderately severe) did not reach statistical significance, caution is warranted in drawing conclusions based on these data.

To better understand the responses to the three enhancements, we further explored the level of engagement and enjoyment ratings for each one.

Web-based, self-guided interventions hold significant promise for people with elevated depressive symptoms. Attaining acceptable levels of adherence to these programs is critical to effectiveness, yet this has proven to be a challenge. We developed and tested 3 enhancements that we hypothesized would, either alone or in combination, improve adherence to a self-guided positive psychological intervention for people with elevated depression. The enhancements were an ODB, VB, and FC. Participants in the eight conditions that received the intervention content also received 1, 2, or all 3 or no enhancements. Our results suggest that the combination of VB and FC is especially impactful and points to areas for future focus to improve adherence across the three enhancements.

For the ODB enhancement, participants had access to an online forum where they could anonymously post questions, share their experiences, and offer encouragement to other participants. Previous findings regarding the efficacy of discussion boards for improving adherence to web-based programs have been mixed. For example, one study found that an internet discussion board or support group resulted in lower adherence to the self-guided program compared with the self-guided program alone [47], although participants who were assigned to receive the discussion board had a greater reduction in depression at 6 months.

The engagement on the discussion board was quite low. Only a quarter of the participants posted on the board at least once, and qualitative feedback indicated that this low level of activity further discouraged participants from making use of the ODB. Therefore, it is perhaps not surprising that there was no indication that the ODB increased adherence to the program, either alone or in combination with other enhancements. Simple availability without engagement is not sufficient for an ODB to increase adherence to the intervention content. There are a number of ways that we could have increased engagement in the ODB, such as requiring participants to post as part of the home practice assignments or making it easier to navigate to the discussion board to lower the barriers to engagement.

VB is a form of gamification and is intended to increase adherence to the intervention content by providing rewards for completing activities on the platform, for example, reaching a goal for the number of log-ins or consecutive homework completions. Participants in the arm that received both VB and FC completed significantly more sessions than did those in the no-enhancement arm (Figure 2). Furthermore, participants in arms that received VB and FC together (VB+FC; VB+FC+ODB) accessed, on average, one more skill (mean 6.42) compared with participants who received either VB or FC without the other (mean 5.42; FC alone, VB alone, ODB+VB, and ODB+FC). Participants were of a mixed opinion on whether the VB were enjoyable, with 33.9% (39/115) giving the VB low enjoyment ratings (below 31 on a 0-100 scale) and 33% (38/115) providing high enjoyment ratings (between 70 and 100). Previous studies document similar challenges to the successful use of gamification elements, such as a lack of clarity on which behaviors are being rewarded or confusion over how progress was being illustrated [48-50]. Future work to make the badges more interpretable, enjoyable, and clearly linked to target behaviors will help to increase the impact of the VB on adherence to the intervention.

Our FC enhancement consisted of staff contacting participants by phone once per week for a brief check-in regarding any challenges they were experiencing while completing that week’s skills and answering questions or concerns they had. We included the FC component based on studies showing that contact with a person improves adherence to self-guided, web-based interventions [18,23,27,51,52]. Although receipt of the FC alone did not significantly improve adherence to the program, when combined with VB, there was an improvement such that those participants who received both FC and VB completed more of the intervention sessions compared with those who did not. The average number of times a facilitator was able to reach a participant was one, and fewer than half of the participants were ever reached by a facilitator by phone. The data indicate that the more participants enjoyed the FC, the more skills they completed, and as with VB, there was a wide range of enjoyment ratings for FC, leaving a lot of room for improvement. One easy change that may increase participants’ enjoyment of the FC condition, and as a result improve the impact of FC on adherence, is to vary the questions that are asked each week and word them so they do not come across as judgmental. Qualitative feedback indicated that participants found these questions repetitive and, especially if they had not completed the home practice that week, somewhat shaming. Furthermore, some participants may prefer email over phone contact, and providing an email contact option could increase the impact of the enhancement. Other ways to improve the impact of the FC condition could include improving the supportive accountability [29], such as setting clear expectations and goals for engagement with the platform, and tailored performance monitoring that is supportive rather than shaming. In addition, clarifying the expertise of the facilitators and increasing trust could enhance the impact of FC [29].

Aside from the content of the enhancement, a simple count of the number of enhancements the participant was assigned to (up to 3) was not predictive of adherence. On the basis of our data, it is not necessarily better. However, the combination of VB and FC increased adherence compared with either enhancement alone. It may be that the FC helps encourage people to engage with the platform, with the phone call providing a reminder and the facilitator providing someone to be accountable. Once participants are on the platform, the VB reinforce that engagement and provide some extrinsic motivation to remain engaged [29].

We explored potential moderators of the effect of each enhancement and found that depression influenced whether VB enhancement was associated with adherence. Specifically, among those with moderately severe baseline depression (PHQ-8 scores between 15 and 19), those who received the VB enhancement demonstrated better adherence than those who were not assigned to a VB condition. For those in the mild and moderate range of the PHQ-8 (scores between 5 and 14), the direction of the effect was the same but did not reach statistical significance. For participants with severe baseline depression (>20), the effect was in the opposite direction, such that VB were associated with (nonsignificantly) poorer adherence. Although this interaction should be interpreted with caution given the marginal significance and the very small number of participants in this group (those with the highest levels of depression who received the VB), this pattern was also apparent in our analyses with the proportion of intervention completed as an outcome (the number of pages viewed as a proportion of the total possible number of pages; Multimedia Appendix 1). It may be that for people with severe clinical depression, the badges were off-putting, and extra thought should be put into this form of gamification for people with the highest levels of depression. Of note, other studies of web-based depression interventions tend to exclude people who have scores in the severe range [10,20] so would not have picked up this potential demotivating effect of VB for people with the highest levels of depression.

There are several limitations to consider in this study. First, the sample was recruited from all web-based sources and likely differed from clinic-based samples or those referred by medical providers. The sample was predominantly made up of non-Hispanic White individuals and females, which significantly limits the generalizability of the findings. Furthermore, participants needed to complete 4 of 7 daily emotion reports as part of a run-in to be assigned to a condition in which only 58.05% (602/1037) of the participants were able to accomplish. Although run-in periods are often used in clinical trials [53], by including only those who demonstrated adherence to the study protocol, we may have artificially enriched our sample with participants who were more likely to be adherent to the intervention. Comparison of those who completed the run-in to those who did not indicate that those who completed were older, more likely to be female, and less depressed than those who consented but did not complete the run-in. We also ran a large number of statistical tests and did not control for multiple comparisons. However, our analyses were hypothesis-driven and appropriate for this stage of intervention development. Finally, we tested the enhancements in the context of a positive psychological intervention, and the content may have influenced adherence, although we do not have a way to test this possibility in this study.

These findings lead to several suggestions for future self-guided interventions and potential adherence enhancements. First, the program and enhancements were feasible even for people with very high levels of depression. Unlike most previous studies of web-based programs for people with depression, we included participants with the highest levels of depression. The findings demonstrate that people with severe levels of depression can also engage in programs like this, and their inclusion in the study sample suggested important differences in terms of response to enhancements. Second, the combination of VB and FC was associated with better adherence than either enhancement alone. Researchers should go beyond single enhancements to improve adherence and consider including combinations of enhancements that may be more effective than any one enhancement alone. Furthermore, it may be helpful to offer participants a choice of which enhancements they want to access. Providing participant control over which enhancements they receive from the outset could increase feelings of engagement and investment in the program. Finally, as the technology for intervention delivery progresses, there are new possibilities for better-engaging participants. Future work may consider more immersive approaches such as virtual reality [54] and approaches such as SMS text messages [55] or chatbots [56] that foster a sense of social connection and may encourage stronger engagement with the intervention content.

Self-guided web-based interventions hold great promise to help people living with depression, and creative approaches to better engage participants will provide greater benefit to more people as the programs are more widely disseminated. Despite low levels of engagement with the FC enhancement, the results of our randomized trial suggest that the combination of VB and FC may be especially effective at improving adherence to the intervention, which in turn, may increase the impact of the intervention on well-being and bode well for future programs that incorporate enhancements such as these.