This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Digital interventions have been effective in improving numerous health outcomes and health behaviors; furthermore, they are increasingly being used in different health care areas, including self-management of long-term conditions, mental health, and health promotion. The full potential of digital interventions is hindered by a lack of user engagement. There is an urgent need to develop effective strategies that can promote users’ engagement with digital interventions. One potential method is the use of technology-based reminders or prompts.
To evaluate the effectiveness of technology-based strategies for promoting engagement with digital interventions.
Cochrane Collaboration guidelines on systematic review methodology were followed. The search strategy was executed across 7 electronic databases: the Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE, Embase, Web of Science, the Education Resources Information Center (ERIC), PsycINFO, and the Cumulative Index to Nursing and Allied Health Literature (CINAHL). Databases were searched from inception to September 13, 2013, with no language or publication type restrictions, using three concepts: randomized controlled trials, digital interventions, and engagement. Gray literature and reference lists of included studies were also searched. Titles and abstracts were independently screened by 2 authors, then the full texts of potentially eligible papers were obtained and double-screened. Data from eligible papers were extracted by one author and checked for accuracy by another author. Bias was assessed using the Cochrane risk of bias assessment tool. Narrative synthesis was performed on all included studies and, where appropriate, data were pooled using meta-analysis. All findings were reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines.
A total of 14 studies were included in the review with 8774 participants. Of the 14 studies, 9 had sufficient data to be included in the meta-analyses. The meta-analyses suggested that technology-based strategies can potentially promote engagement compared to no strategy for dichotomous outcomes (relative risk [RR] 1.27, 95% CI 1.01-1.60, I2=71%), but due to considerable heterogeneity and the small sample sizes in most studies, this result should be treated with caution. No studies reported adverse or economic outcomes. Only one study with a small sample size compared different characteristics; the study found that strategies promoting new digital intervention content and those sent to users shortly after they started using the digital intervention were more likely to engage users.
Overall, studies reported borderline positive effects of technology-based strategies on engagement compared to no strategy. However, the results have to be interpreted with caution. More research is needed to replicate findings and understand which characteristics of the strategies are effective in promoting engagement and how cost-effective they are.
Digital interventions (DIs) are programs that provide information and support—emotional, decisional, and/or behavioral—for physical and/or mental health problems via a digital platform (eg, website or computer) [
In a three-round systematic Delphi experiment done by Brouwer et al [
To our knowledge, none of those reviews has focused specifically on the relationship between engagement, prompts, and the characteristics of prompts. Characteristics likely to influence effectiveness include timing (ie, when should a prompt be used), duration (ie, for how long should it be used) [
A review of digital interventions found that those that used more behavior change techniques (BCTs) were more effective than those that used fewer BCTs [
The aim of this systematic review was to evaluate the effectiveness of technology-based strategies, defined in this review as digital and analog technology methods used to promote the user’s regular interaction with all or part of the DI. These include, but are not limited to, emails, text messages, multimedia messages, telephone calls, automated voice calls, or faxes. Specific objectives of the review were to (1) describe technology-based strategies to promote engagement with DIs, (2) assess the effectiveness of technology-based strategies in promoting engagement with DIs, (3) explore whether different characteristics such as timing, duration, frequency, mode of delivery, sender, content, or use of theory are associated with differential effectiveness, and (4) to describe the cost of technology-based strategies to promote engagement with DIs.
This review followed Cochrane methodological guidance for systematic reviews [
The search was performed in 7 electronic databases: the Cochrane Central Register of Controlled Trials (CENTRAL), MEDLINE, Embase, Web of Science, the Education Resources Information Center (ERIC), PsycINFO (including studies and dissertation abstracts), and the Cumulative Index to Nursing and Allied Health Literature (CINAHL). Databases were searched from inception to September 13, 2013, with no language or publication type restrictions, using three concepts: randomized controlled trials (RCTs)
All citations identified by the search strategy were deduplicated and downloaded into Endnote X5 (Thomson Reuters). Titles and abstracts were screened by one author (GA) and were double-screened by one of 3 other coauthors (EM, FH, or RW). Full texts of potentially eligible articles were screened by 2 authors (EM and GA). Any disagreement was resolved through discussion, referencing the eligibility criteria. If consensus could not be achieved, a third author (FH) was consulted. Justifications for exclusion were recorded and tabulated. All reviewers had training in systematic review methodology.
Participants were adults aged 18 years old or over. There were no limitations on gender, socioeconomic status, ethnicity, or health status. All settings were included for digital intervention; for technology-based strategies, the setting was online.
The interventions of interest were technology-based strategies to promote engagement with digital interventions. To be included, the interventions had to meet the following definitions:
1. Digital interventions were defined as programs that provide information and support—emotional, decisional, and/or behavioral—for physical and/or mental health problems via a digital platform (eg, a website or a computer) [
2. Technology-based engagement-promoting strategies were defined as digital and analog technology methods used to promote the user’s regular interaction with all or part of the DI, including, but not limited to, telephones calls, text messages, multimedia messages, emails, automated voice calls, or faxes. Examples of interventions that were included were a computerized treatment program with mobile phone text messages that reminded the user to visit the program, and a blood pressure self-monitoring website that sent email prompts to users to enter their pressure readings on the website.
Three groups of comparators were defined: (1) minimal or inactive comparators, such as no strategy, (2) nontechnological strategies, such as printed materials or face-to-face contact, and (3) alternative technology-based strategies, for example, where the effects of email prompts are compared to the effects of text message prompts. Some studies tested the cumulative effect of multiple strategies; for example, both arms received prompts by email with one arm also receiving additional prompts by telephone call.
The primary outcome was engagement with the DI, which was recorded as the number of log-ins/visits, number of pages visited, number of sessions completed, time spent on the DI, and number of DI components/features used. These measures were determined in advance before screening included studies [
Two types of secondary outcomes were selected:
1. Adverse outcomes, such as users feeling frustrated or irritated by email prompts, or experiencing a loss of self-esteem due to not being able to engage with the DI.
2. Economic outcomes, which were costs associated with strategies promoting engagement to inform future cost-effectiveness analysis.
RCTs were included; these were either trials of DIs that used strategies promoting engagement or trials evaluating strategies specifically. Economic evaluations were to be included if they were conducted alongside the main trial.
The following were the exclusion criteria:
1. Interventions targeted exclusively at health professionals (eg, computer-based decision aids to assist health professionals in making decisions with regard to treatments).
2. Trials where attrition from the trial and disengagement from the DI are nondistinguishable.
3. Trials where the effect of the DI components cannot be separated from the effect of the engagement-promoting strategy (eg, trials where the DI is not compared to another DI, such as a website to lose weight with email prompts compared with dietician face-to-face sessions with emails from the dietician; or when the difference between the 2 arms included different DIs as well as differential engagement strategies).
In the protocol, it was stated that quasi-RCTs would be included; however, upon further reflection, and due to the reasonable number of eligible RCTs and the high risk of bias associated with quasi-RCTs, they were excluded.
Data were extracted from included papers using an adapted version of the Cochrane Consumers and Communication Review Group data extraction template. One author (GA) extracted all the included papers and another coauthor (FH) verified the accuracy of the extraction; any disagreement was resolved through discussion. If no agreement was reached, a third author (EM) was consulted. Authors were contacted for more information about the characteristics of the strategy and any missing outcome data. The taxonomy for the BCTs [
An assessment of risk of bias was done based on the Cochrane risk of bias assessment tool [
1. Was the allocation sequence adequately generated?
2. Was allocation adequately concealed?
3. Was knowledge of the allocated interventions adequately prevented during the study (ie, blinding)?
4. Were incomplete outcome data adequately addressed?
5. Were study reports free of suggestion of selective outcome reporting?
6. Was the study free of other problems that could put it at risk of bias? These problems included, but were not limited to, baseline characteristic differences between groups, validity and reliability of outcome measures, sample size, and power.
The papers [
Outcome measures were categorized as dichotomous or continuous engagement outcomes:
1. Dichotomous engagement outcome: any dichotomous measure of how participants engaged with the DI, such as proportion of participants who visited the DI, or proportion of participants who completed a prespecified number of modules.
2. Continuous engagement outcome: any continuous measure of how participants engaged with the DI, such as number of visits or page views.
Even within the categories of dichotomous and continuous outcomes, authors often reported more than one outcome. After discussion with coauthors and for the purpose of analysis, one outcome was selected based on the following prespecified criteria:
1. The number of participants who visited the DI (ie, logged in to the website) or the number of visits/log-ins was selected, as these are the most appropriate indicators for engagement strategies [
2. The primary outcome defined or stated by the author.
3. The outcome reported separately for the control and intervention group, rather than lumped together.
4. The highest standard for engagement (ie, the authors report the number of participants who completed all the sessions rather than the number of participants who completed no sessions or a specific number of sessions).
5. Data from the longest measured follow-up period were chosen, as it is important to demonstrate sustained change.
Results were reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [
A meta-analysis was performed and continuous and dichotomous data from RCTs were pooled separately using a random effects model. The appropriate effect measures were determined depending on the type of data. For dichotomous outcomes, relative risks (RRs) and their 95% confidence intervals were used. For continuous outcomes, standardized mean differences (SMDs) with 95% confidence intervals were used. Due to the variable nature of the interventions, heterogeneity was expected and it was assessed using the I2 statistic.
A sensitivity analysis was intended to be undertaken, as recommended by the Cochrane handbook, by excluding trials of poor quality to determine their effects on the study results, as well as a funnel plot to assess publication bias. However, there were insufficient studies to allow for a meaningful assessment. To investigate heterogeneity, a post hoc sensitivity analysis was conducted by removing one study [
Searching the electronic databases yielded a total of 18,881 records. After removing all duplicates (manually and using Endnote X5), 10,133 records remained for title and abstract screening. Of these, 93 went forward for full-text assessment, supplemented by 3 studies identified from reference tracking. A total of 77 papers were excluded at full-text screening for various reasons, the most common being that the engagement strategy or DI did not meet the definition in this review, or that engagement was not measured in the study. There were 4 ongoing studies with only protocols available, and one study was a conference abstract.
PRISMA flow diagram.
A total of 14 studies with 8774 participants were included in the systematic review; their characteristics are described in
All of the studies were conducted online and some studies specifically mentioned the location of the participants: the Netherlands [
Characteristics of included studies.
Study | Digital intervention | Study design, engagement strategy, and comparator |
Berger et al |
Internet-based self-help guide targeting social phobia | Three-arm RCTa (75 participants included in review) |
Berger et al |
Internet-based self-help program targeting depression | Three-arm RCT (50 participants included in review), one arm excluded |
Clarke et al |
Pure self-help program targeting depression | Three-arm RCT (155 participants included in review), one arm excluded |
Couper et al |
Tailored Web program targeting health promotion (ie, intake of fruits and vegetables) | Three-arm RCT (1677 participants included in review), one arm excluded |
Farrer et al |
Web intervention targeting depression | Four-arm RCT (83 participants included in review), 2 arms excluded |
Greaney et al |
Website targeting self-monitoring of physical activity, red meat intake reduction, fruit and vegetable consumption, daily multivitamin use, and smoking cessation | Two-arm RCT and one nonrandomized arm excluded (86 participants included in review) |
McClure et al |
Internet intervention targeting smoking cessation | Randomized factorial trial (1865 participants included in review) |
Muñoz et al |
Web-based intervention targeting smoking cessation | Four-arm RCT (498 participants included in review), 2 arms excluded |
Proudfoot |
Online psychoeducation program targeting bipolar disorder | Three-arm RCT (273 participants included in review), one arm excluded |
Santucci et al |
An entirely automated and tailored Web-based intervention targeting anxiety and depression | Two-arm RCT (43 participants included in review) |
Schneider |
Computer-tailored program targeting multiple health behaviors: physical activity, fruit and vegetable intake, smoking cessation, and decreasing alcohol consumption | Two-arm RCT (3448 participants included in review) |
Schneider |
Internet-delivered computer-tailored program targeting multiple health behaviors: physical activity, fruit and vegetable intake, smoking cessation, and decreasing alcohol consumption | Seven-arm RCT (240 participants included in review) |
Simon et al |
An interactive online program targeting bipolar disorder | Two-arm RCT (118 participants included in review) |
Titov et al |
A computer-delivered treatment targeting social phobia | Two-arm RCT (163 participants included in review) |
aRCT: randomized controlled trial.
The digital interventions targeted different health behaviors and conditions. Eight DIs were designed to target different mental health conditions, including social phobia [
Four studies used their strategies at different time points. One engagement strategy was used at weeks 2 and 3 from baseline [
Strategies were used either for the duration of the DI [
Most of the studies reported using engagement strategies on a regular basis. Six studies used the strategy at least once per week [
Email was the most commonly used mode of delivery among the different studies [
Other characteristics that were identified were the type of sender or provider and whether the strategies were automated [
The content of the strategies was classified into 5 types: offering assistance with the DI [
No paper provided information about any underlying theoretical framework for the use, delivery, or content of strategies.
Tailoring was reported in 3 studies. In one study, participants received reports about the frequency of their usage of the DI via emails [
The studies differed in the way they were conducted and some did not provide sufficient information to judge their quality. All studies reported randomization but only 9 reported adequate sequence generation process [
Data suitable for meta-analysis were only available for the comparison of a technology-based engagement strategy with no strategy. Two meta-analyses were performed, using dichotomous and continuous outcomes. The outcome measures of the studies included in the meta-analyses were number of DI modules/sessions/lessons completed, number of participants who completed DI modules/sessions/lessons, and number of participants who logged in/visited the DI; the outcome measures for the rest of the studies can be found in
Eight studies with 6120 participants reported sufficient data to be included in the meta-analyses, comparing a technology-based engagement strategy to no strategy using dichotomous outcomes (Analysis 1.1) (see
Analysis 1.1. Technology-based engagement strategy compared to no strategy: dichotomous outcomes.
Analysis 1.2. Technology-based engagement strategy compared to no strategy: continuous outcomes.
For the other comparator types, for which a meta-analysis was not performed, one study compared technology-based engagement strategies to nontechnological means of engagement (ie, comparing telephone calls to postal mail). The postal mail group had an average of 5.9 visits and the telephone call group had an average of 5.6 visits (mean difference = 0.3 visits,
As for the multiple strategies group, 3 studies had 2 arms with the same technology-based engagement strategy and one of the arms received an extra strategy delivered through telephone calls. None of the studies reported a significant difference in the effect of using multiple strategies on engagement [
No conclusions can be drawn about the effect of the different characteristics, as only one study compared the effects of timing and content of strategies on engagement with a DI. The study found that strategies sent early and those that showed DIs' updated content were more likely to engage users [
Data on adverse and economic outcomes were intended to be extracted; however, none of the included studies reported these outcomes.
All authors were contacted to provide and confirm information about missing or unclear engagement outcome information or characteristics of strategies, and 4 authors replied. Farrer et al provided the mean and standard deviation of BluePage visits and time spent, and more information about the strategy, including the fact that it was not tailored [
Technology-based strategies to promote engagement are an emerging field of research as shown by the number of included studies and their dates of publication. Generally, studies report borderline small-to-moderate positive effects of technology-based strategies on engagement compared to using no strategy, which support the use of technological strategies to improve engagement. However, this result should be treated with caution due to the high heterogeneity, small sample sizes, and the lack of statistical significance in the analysis of continuous outcomes. There were insufficient studies to effectively explore reasons for heterogeneity. No firm conclusions were drawn about which characteristics of strategies were associated with effectiveness, and due to the absence of data, no conclusions could be drawn about costs or cost-effectiveness. Although the review aimed to investigate the cost-effectiveness of engagement strategies, none of the included papers reported cost data.
To our knowledge, this is the first systematic review that evaluated technology-based engagement-promoting strategies, using website metrics as outcome measures. Other systematic reviews [
The findings in this review agree with previous reviews that technology-based strategies may potentially promote engagement, but that there is substantial heterogeneity, potentially due to the different outcome measures used [
Authors often report multiple measures of engagement, and these often vary between studies. As measures of engagement are likely to vary depending on the research question, characteristics of the engagement strategy, and the DI, clear guidance for the optimal reporting of engagement is urgently needed. Researchers need to describe and detail clearly how a DI is intended to achieve its outcomes, the level of engagement intended or desired, and the rationale for that. For example, consider a structured and session-based DI targeting a mental disorder with an email prompting users to complete all the sessions to benefit from the DI, and the research question measuring how many participants completed all the sessions—an appropriate engagement measure would be the number of participants completing all the sessions rather than number of visits or time spent on the DI.
Authors should also clearly define their concept of optimal engagement in future studies, specifying a primary outcome for engagement and the rationale for choosing it. This is supported by the fact that the other systematic reviews of engagement reported that one of the most common reasons for excluding studies is a lack of reported engagement outcomes [
This review identified themes in terms of characteristics of strategies to enable future research to selectively evaluate the different characteristics. Future primary studies that aim to determine the effectiveness of technological strategies on engagement with DIs should include a detailed description of the characteristics of engagement strategies, specifically the content of these strategies, and whether using different BCTs influence effectiveness. For this description, researchers could use the categories in this review, or expand on them. Researchers should also report the context (eg, characteristics of the DI) and outcome measures that contribute to heterogeneous results. This can help when conducting meta-analyses of future systematic reviews [
Researchers should also differentiate between attrition from the trial (ie, dropout attrition or loss to follow-up) and disengagement from the DI (ie, nonusage attrition), because studies have shown that the relation between these different types of attrition are complex and they do not share the same associated factors [
The main strengths of this review are the rigorous and systematic methodology, which followed Cochrane methodological guidance, and the comprehensive and extensive search strategy. Furthermore, screening, extraction, and risk of bias assessment were independently conducted or reviewed by at least two authors. The review also includes meta-analyses to measure the effect of using the strategies compared to no strategies. In addition, the published, peer-reviewed protocol provides transparency.
The systematic review included RCTs as the most rigorous method for evaluating strategies, however, it is increasingly being recognized that the inclusion of other types of research is important. Policy makers and researchers are facing complex questions that the rigid and quantitative types of studies might not answer most appropriately. Rather, qualitative studies might be more equipped to fill in the gaps that RCTs cannot provide an answer for, such as the experiences of participants, the possible contradiction in some outcomes, and theory development [
The limited search of the grey literature might be considered a limitation; however, in the case of this emerging field of research, the risk of significant publication bias is probably low because both negative and positive findings are of interest. A funnel plot could have been used to estimate the degree of publication bias; however, this was not possible because of the low number of studies, and the possibility of funnel plot asymmetry due to the different methodological qualities of the studies regardless of the existence of publication bias [
Technology-based strategies may promote engagement compared to using no strategy; however, this finding should be interpreted with caution as only a small number of eligible studies were identified for the meta-analysis and the results were heterogeneous. The field of engagement strategies is an emerging field, as indicated by the number and dates of the studies; more research is needed to understand what strategy characteristics are effective and how cost-effective they are.
MEDLINE search strategy.
(A) Risk of bias summary. (B) Risk of bias graph.
Comparison engagement strategy versus no engagement strategy: dichotomous outcomes sensitivity analysis after removing the study by Schneider et al [
Detailed characteristics of included studies.
Main engagement outcomes and findings reported in included studies.
behavior change technique
Cochrane Central Register of Controlled Trials
Cumulative Index to Nursing and Allied Health Literature
digital intervention
Education Resources Information Center
Journal of Medical Internet Research
Preferred Reporting Items for Systematic Reviews and Meta-Analyses
randomized controlled trial
relative risk
standardized mean difference
University College London
Many thanks go to Ruth Muscat, Knowledge Resources Librarian, from University College London (UCL), London, UK, who assisted with developing the search strategy, and to Kate Sheals, Research Assistant, Department of Clinical, Educational and Health Psychology, University College London, London, UK, who was consulted on the BCT coding. GA is a PhD student at UCL, funded by the Saudi Cultural Bureau.
None declared.