The search was performed in 7 electronic databases: the Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE, Embase, Web of Science, the Education Resources Information Center (ERIC), PsycINFO (including studies and dissertation abstracts), and the Cumulative Index to Nursing and Allied Health Literature (CINAHL). Databases were searched from inception to September 13, 2013, with no language or publication type restrictions, using three concepts: randomized controlled trials (RCTs) and digital interventions and engagement (see Multimedia Appendix 1 for the MEDLINE search strategy). The search also included screening grey literature (Conference Proceedings Citation Index, formerly ISI Proceedings), references of the included studies, issues of key journals such as the Journal of Medical Internet Research (JMIR), and using Google Scholar to screen any papers citing included or other key papers [18,20,22,23].

All citations identified by the search strategy were deduplicated and downloaded into Endnote X5 (Thomson Reuters). Titles and abstracts were screened by one author (GA) and were double-screened by one of 3 other coauthors (EM, FH, or RW). Full texts of potentially eligible articles were screened by 2 authors (EM and GA). Any disagreement was resolved through discussion, referencing the eligibility criteria. If consensus could not be achieved, a third author (FH) was consulted. Justifications for exclusion were recorded and tabulated. All reviewers had training in systematic review methodology.

The following were the exclusion criteria:

1. Interventions targeted exclusively at health professionals (eg, computer-based decision aids to assist health professionals in making decisions with regard to treatments).

2. Trials where attrition from the trial and disengagement from the DI are nondistinguishable.

3. Trials where the effect of the DI components cannot be separated from the effect of the engagement-promoting strategy (eg, trials where the DI is not compared to another DI, such as a website to lose weight with email prompts compared with dietician face-to-face sessions with emails from the dietician; or when the difference between the 2 arms included different DIs as well as differential engagement strategies).

In the protocol, it was stated that quasi-RCTs would be included; however, upon further reflection, and due to the reasonable number of eligible RCTs and the high risk of bias associated with quasi-RCTs, they were excluded.

Data were extracted from included papers using an adapted version of the Cochrane Consumers and Communication Review Group data extraction template. One author (GA) extracted all the included papers and another coauthor (FH) verified the accuracy of the extraction; any disagreement was resolved through discussion. If no agreement was reached, a third author (EM) was consulted. Authors were contacted for more information about the characteristics of the strategy and any missing outcome data. The taxonomy for the BCTs [32] was used; strategy contents were coded by one author (GA) during data extraction and verified by another author (RW), who is an experienced user of the taxonomy.

An assessment of risk of bias was done based on the Cochrane risk of bias assessment tool [33]. The following criteria were used:

1. Was the allocation sequence adequately generated?

2. Was allocation adequately concealed?

3. Was knowledge of the allocated interventions adequately prevented during the study (ie, blinding)?

4. Were incomplete outcome data adequately addressed?

5. Were study reports free of suggestion of selective outcome reporting?

6. Was the study free of other problems that could put it at risk of bias? These problems included, but were not limited to, baseline characteristic differences between groups, validity and reliability of outcome measures, sample size, and power.

The papers [11,35-47] were categorized as having low, high, or unclear risk of bias (ie, when the study did not provide enough information to judge the different aspects of trial quality). A risk of bias summary (see Multimedia Appendix 2A) and a risk of bias graph (see Multimedia Appendix 2B) were generated. The bias assessment was done by one author (GA) and was checked by another author (FH). Any discrepancies were resolved by a third author (EM).

Searching the electronic databases yielded a total of 18,881 records. After removing all duplicates (manually and using Endnote X5), 10,133 records remained for title and abstract screening. Of these, 93 went forward for full-text assessment, supplemented by 3 studies identified from reference tracking. A total of 77 papers were excluded at full-text screening for various reasons, the most common being that the engagement strategy or DI did not meet the definition in this review, or that engagement was not measured in the study. There were 4 ongoing studies with only protocols available, and one study was a conference abstract. Figure 1 shows the results of the initial searches, screening, and selection processes.

A total of 14 studies with 8774 participants were included in the systematic review; their characteristics are described in Table 1, with full details shown in Multimedia Appendix 4. The sample sizes ranged from 43 to 3448. One study was published in 2005 and the rest were published between 2009 and 2013; all studies were published in English. More than half of the studies [11,35,36,39-43] had more than 2 arms, but, with the exception of one study [39], only 2 arms met the inclusion criteria (see Table 1). One study was a factorial RCT where half of the participants received an engagement strategy while the other half did not [37]. One study had 7 arms assessing the effect of different timing and content of strategies [44]. The remaining 4 studies were RCTs with 2 arms [38,45-47].

All of the studies were conducted online and some studies specifically mentioned the location of the participants: the Netherlands [44,46], Australia [36,43,47], the United States [11,35,37,38,41], and Switzerland and Germany [40]. Six of the studies aimed to evaluate the effect of adding the strategy on the effectiveness of the DIs [35,39-42,47], 3 of the studies aimed to evaluate the effect of the technology-based strategies on promoting engagement with the DI [37,45,46], and 2 studies aimed to evaluate the effect of the strategy on digital intervention outcomes and engagement [36,43]. One study evaluated the effect of different timing and content of strategies on engagement [44], one study evaluated the effect of adding online peer coaching on increasing participation with a DI [38], and the final study explored the qualities of engagement with a DI [11].

The digital interventions targeted different health behaviors and conditions. Eight DIs were designed to target different mental health conditions, including social phobia [39,47], depression [35,36,40,45], anxiety [45], and bipolar disorder [38,43]. The rest of the DIs promoted a variety of health behaviors, including smoking cessation (n=4), decreasing alcohol consumption (n=2), self-monitoring of healthy behaviors (n=1), physical activity (n=2), and healthy diet (n=3) [11,37,41,42,44,46]. Most of the studies included detailed descriptions of the DIs. Two DIs were described as self-help guides with modules presented in a sequential order and participants could complete the whole program at once or over time [39,40]. Six DIs were composed of sessions that were presented in a sequential and phased order [11,36,43-46]. There were 2 studies that updated their DIs with new information [44,46], and 2 described their DIs as interactive [35,38].

The studies differed in the way they were conducted and some did not provide sufficient information to judge their quality. All studies reported randomization but only 9 reported adequate sequence generation process [35,37,39,40,42-44,46,47]. Ten studies had adequate allocation concealment [35-37,39,40,42-44,46,47]. One study reported that participants and researchers were blinded [43]. Engagement measures were prespecified in 11 studies [11,36-38,40-46], however, 3 studies out of these did not report some engagement outcomes for the intervention and control group separately [11,37,42]. Engagement measures were measured objectively, so no bias was identified for any of the studies in terms of incomplete outcome data except for one study where engagement measures were not reported for 6 participants who dropped out [39]. Protocols were only reported in 3 studies [36,37,46].

Data on adverse and economic outcomes were intended to be extracted; however, none of the included studies reported these outcomes.

All authors were contacted to provide and confirm information about missing or unclear engagement outcome information or characteristics of strategies, and 4 authors replied. Farrer et al provided the mean and standard deviation of BluePage visits and time spent, and more information about the strategy, including the fact that it was not tailored [36]. McClure et al provided the exact number of people allocated to the strategy and the fact that the strategy was used for 12 months [37]. Clarke et al [35] and Simon et al [38] both confirmed the accuracy in categorizing their strategies’ characteristics.

Technology-based strategies to promote engagement are an emerging field of research as shown by the number of included studies and their dates of publication. Generally, studies report borderline small-to-moderate positive effects of technology-based strategies on engagement compared to using no strategy, which support the use of technological strategies to improve engagement. However, this result should be treated with caution due to the high heterogeneity, small sample sizes, and the lack of statistical significance in the analysis of continuous outcomes. There were insufficient studies to effectively explore reasons for heterogeneity. No firm conclusions were drawn about which characteristics of strategies were associated with effectiveness, and due to the absence of data, no conclusions could be drawn about costs or cost-effectiveness. Although the review aimed to investigate the cost-effectiveness of engagement strategies, none of the included papers reported cost data.

To our knowledge, this is the first systematic review that evaluated technology-based engagement-promoting strategies, using website metrics as outcome measures. Other systematic reviews [21-23] investigated the effect of technological engagement strategies of DIs on behavior change and some looked at engagement-promoting features of DIs, including the use of emails and telephone calls on the change in website metrics [25]. All of these systematic reviews reported a potentially positive effect of engagement strategies on changing health behavior and engagement. However, Brouwer et al, who used similar outcome measures, did not do a meta-analysis due to the heterogeneity of the outcome measures [25].

The findings in this review agree with previous reviews that technology-based strategies may potentially promote engagement, but that there is substantial heterogeneity, potentially due to the different outcome measures used [16,25,50,51], characteristics of the DI, and engagement strategies. In this systematic review, the measures were categorized into continuous and dichotomous outcomes, and outcomes were selected for meta-analysis using prespecified criteria. This allowed for performing two meta-analyses that shared similar studies but different measures. The two meta-analyses showed a similar direction of effect.

Authors often report multiple measures of engagement, and these often vary between studies. As measures of engagement are likely to vary depending on the research question, characteristics of the engagement strategy, and the DI, clear guidance for the optimal reporting of engagement is urgently needed. Researchers need to describe and detail clearly how a DI is intended to achieve its outcomes, the level of engagement intended or desired, and the rationale for that. For example, consider a structured and session-based DI targeting a mental disorder with an email prompting users to complete all the sessions to benefit from the DI, and the research question measuring how many participants completed all the sessions—an appropriate engagement measure would be the number of participants completing all the sessions rather than number of visits or time spent on the DI.

Authors should also clearly define their concept of optimal engagement in future studies, specifying a primary outcome for engagement and the rationale for choosing it. This is supported by the fact that the other systematic reviews of engagement reported that one of the most common reasons for excluding studies is a lack of reported engagement outcomes [19,25]. Another issue related to engagement measures is the extent/duration or level of engagement that defines whether a user is successfully engaging with a DI or not. One attempt to quantify engagement was done by Kelders et al in a systematic review, which stated that a typical DI will have 50% of users engaged in it, using it at least once a week and up to 10 weeks. More research is needed to identify whether an outcome such as duration/level of engagement is enough to produce a positive effect size that justifies the cost of developing and implementing DIs [19].

This review identified themes in terms of characteristics of strategies to enable future research to selectively evaluate the different characteristics. Future primary studies that aim to determine the effectiveness of technological strategies on engagement with DIs should include a detailed description of the characteristics of engagement strategies, specifically the content of these strategies, and whether using different BCTs influence effectiveness. For this description, researchers could use the categories in this review, or expand on them. Researchers should also report the context (eg, characteristics of the DI) and outcome measures that contribute to heterogeneous results. This can help when conducting meta-analyses of future systematic reviews [52]. In addition, researchers should report multiple measures of outcome over the duration of the DI and not only report the engagement measure postintervention.

Researchers should also differentiate between attrition from the trial (ie, dropout attrition or loss to follow-up) and disengagement from the DI (ie, nonusage attrition), because studies have shown that the relation between these different types of attrition are complex and they do not share the same associated factors [18,20]. Disengagement is likely to impact on the effectiveness of the DI. It may be related to characteristics of the intervention (eg, design, usability, and perceived effectiveness) or to characteristics of the user (eg, motivation, self-efficacy, and resources). Loss to follow-up affects the ability of the study to answer the research question posed, with poor follow-up rates negatively impacting both the precision and the robustness of any estimate of effect.

The main strengths of this review are the rigorous and systematic methodology, which followed Cochrane methodological guidance, and the comprehensive and extensive search strategy. Furthermore, screening, extraction, and risk of bias assessment were independently conducted or reviewed by at least two authors. The review also includes meta-analyses to measure the effect of using the strategies compared to no strategies. In addition, the published, peer-reviewed protocol provides transparency.

The systematic review included RCTs as the most rigorous method for evaluating strategies, however, it is increasingly being recognized that the inclusion of other types of research is important. Policy makers and researchers are facing complex questions that the rigid and quantitative types of studies might not answer most appropriately. Rather, qualitative studies might be more equipped to fill in the gaps that RCTs cannot provide an answer for, such as the experiences of participants, the possible contradiction in some outcomes, and theory development [53]. In the case of engagement, certain issues can only be answered through conducting qualitative studies rather than quantitative ones [54]. These issues may include understanding what outcomes mean for the user (eg, DI visits, page views, and time spent on the DI), what the experience of the engaged user is compared to the disengaged user, and the preference of users.

The limited search of the grey literature might be considered a limitation; however, in the case of this emerging field of research, the risk of significant publication bias is probably low because both negative and positive findings are of interest. A funnel plot could have been used to estimate the degree of publication bias; however, this was not possible because of the low number of studies, and the possibility of funnel plot asymmetry due to the different methodological qualities of the studies regardless of the existence of publication bias [33]. Another possible limitation might be that the use of the current Cochrane bias assessment guidelines might be more suitable for generic drug trials as opposed to DIs. For example, sequence generation is not an issue as judged in this review, as it is made easier with the use of online randomization programs. Blinding of staff and participants might not be possible as the control and intervention groups may be aware of receiving strategies sent by the staff. Criteria for traditional outcome assessment might not be suitable for reviewing studies of engagement, as it has to be tailored to how engagement is measured (eg, by automatic website metrics). For most of the studies, the description provided was not sufficient to judge the different aspects of trial quality. Authors and developers of DIs can benefit from using the enhanced CONSORT-EHEALTH reporting guide, published by JMIR. It can help clarify what authors need to report and describe in their studies to enable readers and reviewers to judge a study’s quality [55].

Technology-based strategies may promote engagement compared to using no strategy; however, this finding should be interpreted with caution as only a small number of eligible studies were identified for the meta-analysis and the results were heterogeneous. The field of engagement strategies is an emerging field, as indicated by the number and dates of the studies; more research is needed to understand what strategy characteristics are effective and how cost-effective they are.