This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Adherence reflects the extent to which individuals experience or engage with the content of online interventions and poses a major challenge. Neglecting to examine and report adherence and its relation to outcomes can compromise the interpretation of research findings.
The aim of this systematic review is to analyze how adherence is accounted for in publications and to propose standards for measuring and reporting adherence to online interventions.
We performed a systematic review of randomized controlled trials on online interventions for the prevention and treatment of common mental disorders (depression, anxiety disorders, substance related disorders, and eating disorders) published between January 2006 and May 2018 and indexed in Medline and Web of Science. We included primary publications on manualized online treatments (more than 1 session and successive access to content) and examined how adherence was reported in these publications.
We identified 216 publications that met our inclusion criteria. Adherence was addressed in 85% of full-text manuscripts, but only in 31% of abstracts. A median of three usage metrics were reported; the most frequently reported usage metric (61%) was intervention completion. Manuscripts published in specialized electronic health journals more frequently included information on the relation of adherence and outcomes.
We found substantial variety in the reporting of adherence and the usage metrics used to operationalize adherence. This limits the comparability of results and impedes the integration of findings from different studies. Based on our findings, we propose reporting standards for future publications on online interventions.
Online interventions have become popular in the prevention and treatment of mental disorders, and they have been shown to be effective in clinical trials for a wide range of common mental disorders [
However, the behavior changes that online interventions aim to induce are very unlikely to occur if participants expose themselves to the intervention only briefly or do not do so at all. Adherence can be conceptualized as the extent to which individuals experience or engage with the content of an online intervention [
Attention to adherence has increased over time. However, even 13 years after Eysenbach’s landmark paper “The law of attrition” [
If an intervention is not effective even though the participants have been using it the way they were supposed to, it is very likely that the intervention itself is not working and that the core components of the intervention need to be changed or that there is a mismatch between user needs and intervention components. If, however, the intervention is not effective while people are not exposing themselves to a sufficient “dose” of it, implications for further research might be quite different. A mismatch between user needs and the intervention or its components is likely in these cases. Poor adherence may then lead to systematic underestimation of the potential intervention effects. Instead of changing core components of the intervention that teach skills and prompt change in behaviors related to mental health, the intervention may need to be augmented with components to improve adherence, or recruitment strategies may need to be changed to reach those who are open to actually using the interventions. Thus, it is vital to study both intervention effects of and adherence to online interventions and their interactions simultaneously. Furthermore, it is important to identify differential usage patterns in multicomponent interventions and user characteristics that are associated with these patterns [
Although adherence has received increasing attention in the study of online interventions and been addressed in existing reporting guidelines [
In addition to a variety of terms that describe how users adhere to interventions, there are also numerous ways to measure adherence. Most online intervention platforms store log data (eg, time spent on the intervention page), which allow us to track if and how users interact with the intervention. Despite some shortcomings of this data collection method, such as not knowing with certainty whether the person who used the program was the same person who signed up, or whether a user actually engaged with the content or just opened the pages without further engagement, it provides us with objective and comparable usage metrics. However, there is much variety in the usage metrics reported for online interventions (eg, percentage of participants completing all modules/sessions, percentage of participants who visited/revisited websites, average duration of visits, average number of log-ins, and average number of pages visited) [
Michie et al [
Although higher adherence has been shown to be linked to larger intervention effects in numerous trials [
The CONSORT-eHEALTH (Consolidated Standards of Reporting Trials of Electronic and Mobile HEalth Applications and onLine TeleHealth) guidelines [
The aim of our systematic review was to analyze how adherence has been addressed and which usage metrics have been reported in primary publications on randomized controlled trials (RCTs) evaluating manualized, multisession online interventions (including interventions that are delivered through mobile devices) for common mental disorders (ie, depression, anxiety, substance use disorders, and eating disorders). Specifically, we examined whether adherence was reported in the abstract, the results section, and the CONSORT flowchart of each publication; which usage metrics were reported; whether usage or adherence were addressed in the discussion section; and whether usage metrics were analyzed in relation to outcome.
Based on our findings, we propose common standards for addressing adherence, including specific recommendations for usage metrics that the existing CONSORT-eHEALTH guidelines are currently not specifying.
We included articles that met the following criteria in the review: (1) the article was published in a peer reviewed journal after the publication of Eysenbach’s seminal “The Law of attrition” [
We excluded articles that met the following criteria from the review: (1) The article described research on an online intervention targeting common mental disorders in patients with a primary somatic disorder (eg, diabetes or cancer) or an online intervention targeted at carers or parents of patients; (2) the trial examined a highly individualized intervention without common core content; (3) the trial examined a blended intervention, and (4) the trial examined an intervention purely based on text messaging, email, online discussion boards, or online chat groups.
This review has not been preregistered, and the review protocol has not been published.
We conducted a literature search using the Medline and Web of Science databases. We used the following search terms: “online,” “internet,” “webbased,” “mobile”; “treatment,” “psychotherapy,” “therapy,” “self-help,” “prevention,” “intervention”; and “depression,” “depressive,” “anxiety,” “phobia,” “phobic,” “eating disorder,” “disordered eating,” “anorexia,” “anorexic,” “bulimia,” “bulimic,” “binge eating,” “substance abuse,” “substance related disorder,” “alcohol,” “nicotine,” and “cannabis” (
Studies were selected in two steps. First, titles and abstracts were screened by author IB to exclude publications that were clearly out of the scope of the review; did not report studies on interventions targeting common mental disorders; described study protocols, reviews, and meta analyses; reported secondary analyses only; or had not been published in a peer review journal. Second, authors IB, BV, PM, and AZ assessed the remaining full-text articles for eligibility. Each publication was assessed by two authors. We coded the following variables (along the bibliographic data) for each publication: (1) Is this an RCT? (2) Is this an online intervention? (3) Is it manualized? (4) Does it have multiple sequential sessions/modules? (5) Does it teach skills? (6) Is this the main publication? (7) Study registration number.
Data extraction was conducted in three steps. First, authors BV and IB coded the following variables for all included studies: (1) Is adherence addressed in the abstract? (2) Is adherence addressed in the results section? (3) Is adherence addressed in the CONSORT statement? (4) Is adherence examined in relation to outcome? (5) Is adherence addressed in the discussion section?
In the next step, authors BV and IB extracted the information on adherence reported in the results section and the CONSORT statement. For that purpose, a data extraction form was developed, which captured the following usage metrics: full intervention completion (eg, “XX%/N completed the full intervention” or “XX%/N failed to complete the full intervention”), completion of a set minimum of sessions/modules (eg, “XX%/N completed 6 out of 8 sessions” or “XX%/N completed less than 2 out of 5 sessions”), average number of completed sessions/modules, specified point of intervention dropout (last opened session or module, sometimes illustrated by a graph), intervention dropout (not specified, eg, “XX%/N did not complete the intervention”), number of participants who were allocated to the intervention but never logged on, number of times participants logged on, proportion of patients accessing the treatment site per week, total time spent on the program, time spent on the program site per week/per login, and number of participants who completed a survey that is part of the intervention (not just assessment for the clinical trial), number of entries in a diary, number of completed exercises, number of messages to a coach, number of participants who posted to a discussion board, number of participants who accessed a discussion board (without necessarily posting something), number of visits to the discussion board, number of participants who shared diary entries with other participants, average percentage of pages read, and average percentage of screens viewed. The use of this form was piloted and revised in a group meeting (IB, PM, and BV) on a subset of the included articles (N=150). The resulting data extraction form was then used to extract data from the remaining studies (IB, FB, PM, BV, and AZ). Disagreements regarding the coding were discussed between IB and BV until a consensus was reached.
Data were entered into an Excel spreadsheet, which was then converted into an SPSS (Statistical Package for the Social Sciences) file. Each publication was treated as a separate case. Descriptive analyses were performed using SPSS [computer software] (Version 24.0. IBM Corp, Armonk, NY). Absolute and relative frequencies were used as the primary measures for the adherence reporting and defined usage metrics. We tested differences in adherence reporting and usage metrics between studies published in specialized eHealth journals versus nonspecialized journals. Two-sided Chi-squared tests were used, and
We identified 216 publications reporting on the primary outcomes of RCTs investigating online interventions for common mental disorders (
Interventions for depression (n=73) were evaluated most frequently, followed by interventions for anxiety disorders (n=65). Substance use disorders (n=27) and eating disorders (n=24) were targeted less frequently. Transdiagnostic interventions were investigated in 27 trials. The majority of trials evaluated interventions aiming to treat mental disorders (n=177); prevention (n=34) and aftercare (n=5) interventions were less frequently investigated.
Although roughly one in two publications in specialized eHealth journals included information on how adherence was related to outcomes, this was only true for one in four publications in nonspecialized journals (χ21=4.6,
In the 188 publications that contained information on adherence, a median of three usage metrics was reported. Of the total of 216 publications, 23 (10.6%) included one metric, 46 (21.3%) included two metrics, 56 (25.9%) included three metrics, and 63 (29.2%) included four or more metrics. Most metrics were related to session/module progression, while the usage of additional intervention features (eg, diaries, discussion boards, and messaging) was rarely reported. The most frequently reported metric was “full intervention completion” (
Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow diagram. CMHD: common mental health disorders; RCT: randomized controlled trial.
Adherence reporting.
Adherence reporting | Overall (N=216), n (%) | Studies published in eHealtha journals (n=34), n (%) | Studies published in nonspecialized journals (n=182), n (%) | ||
Adherence not addressed at all | 28 (13.0) | 5 (14.7) | 23 (12.6) | .74 | |
Adherence addressed in the abstract | 65 (30.1) | 12 (35.3) | 53 (29.1) | .47 | |
|
180 (83.3) | 29 (85.3) | 151 (83.0) | .74 | |
|
Under own heading | 144 (66.7) | 24 (70.6) | 120 (65.9) | .60 |
|
Adherence related to outcome | 62 (28.7) | 15 (44.1) | 47 (25.8) | .03 |
Adherence addressed in CONSORTb chart | 89 (41.2) | 11 (32.4) | 78 (42.9) | .22 | |
Adherence addressed in discussion | 150 (69.4) | 24 (70.6) | 126 (69.2) | .88 |
aeHealth: electronic health.
bCONSORT: Consolidated Standards of Reporting Trials.
Usage metrics (used in at least 10 publications).
Usage metrics | Overall (N=216), n (%) | Studies published in eHealtha journals (n=34), n (%) | Studies published in nonspecialized journals (n=182), n (%) | |
Full intervention completion | 127 (58.8) | 19 (55.9) | 108 (59.3) | .71 |
Completion of set minimum of sessions/modules | 99 (45.8) | 21 (61.8) | 78 (42.9) | .04 |
Average number of completed sessions/modules | 82 (38.0) | 9 (26.5) | 73 (40.1) | .13 |
Specified point of intervention dropout (last opened/completed session/module) | 71 (32.9) | 12 (35.3) | 59 (32.4) | .74 |
Intervention dropout (point not specified) | 28 (13.0) | 3 (8.8) | 5 (2.7) | .43 |
Number of participants who were allocated to the intervention, but never logged in | 68 (31.5) | 13 (38.2) | 55 (30.2) | .36 |
Number of times participants were logged on | 18 (8.3) | 2 (5.8) | 16 (8.8) | .57 |
Total time spent on program | 21 (9.7) | 0 (0) | 21 (11.5) | .04 |
Number of entries in a diary | 12 (5.6) | 3 (8.8) | 9 (4.9) | .37 |
Number of messages to a coach | 17 (7.9) | 0 (0) | 17 (9.3) | .06 |
aeHealth: electronic health.
The aim of this systematic review was to analyze reporting of adherence to online interventions for common mental disorders and to propose recommendations for reporting adherence in future publications. The majority of publications included information on program usage, but in 13% of the publications, adherence was not referred to at all. Adherence was typically addressed in the results section (often under its own heading), less often in the discussion section, and upfront in the abstract in only in one-third of the publications.
In the majority of publications, multiple usage metrics were reported, which is in line with recommendations given in previous reviews on adherence [
Some usage metrics might be more useful or valid than others in terms of how they reflect actual usage behavior. For example, measuring time spent in the intervention may not be the most appropriate metric, as it is still not trivial to determine whether recordings are related to actual intervention use or some other activity in another tab of the same browser or even outside the browser. Completion of exercises throughout the online intervention, on the other hand, might be a user metric that can capture deeper content-focused engagement with the intervention.
In our review, only one in four publications addressed adherence in relation to outcome; publications from specialized eHealth journals did so significantly more often than publications from nonspecialized journals. A previous review [
Based on our review and previous reviews, we propose reporting standards regarding adherence and usage metrics (
The assessment and report of multiple usage metrics is encouraged, as it has several advantages. First, the use of individual components can be investigated; thus, multiple usage patterns can be identified and ultimately be linked to outcome [
If an intervention includes components like diaries, discussion boards, or messaging tools that are considered an essential part of the intervention by the developers, information on the usage of these components should be provided to allow examination of the actual benefit of the component. Participants in interventions with multiple components may exhibit different usage patterns, and adherence measures should reflect this by including use metrics related to those different components. Analyses of multiple usage metrics can extend our knowledge on the most relevant measures (ie, those closest related to outcome) or parameters a composite score for “overall adherence” should contain.
If possible, adherence should be addressed from two perspectives: progress through the intervention and the level of active engagement with the intervention content. This may help distinguish between users who only “consume” the content (eg, read texts and watch videos) and comply with the protocol and those who actively engage with the intervention (eg, write messages, use diaries, and implement behavior chances). It is, however, a challenge to measure this active engagement, which should therefore be a priority for future research to investigate.
The possible impact of adherence on intervention outcomes should be addressed in the discussion. As appropriate, secondary analyses investigating this impact should be undertaken.
Address adherence in every publication regarding online interventions.
Provide details on how adherence was operationalized and how usage metrics were obtained in the methods section.
Include information on adherence in your abstract.
Provide detailed information on adherence in the results section and the Consolidated Standards of Reporting Trials flow chart:
Include detailed information on user retention per session or module. Avoid only reporting dichotomized data (eg, treatment completers vs noncompleters, low vs high adherers).
Include at least the following metrics: average number of completed modules/session and number of participants who were allocated to the intervention but never logged in.
In interventions with multiple components, include metrics that reflect those different components.
Differentiate between passive components (eg, reading assignments and videos) and active components that require engagement (eg, diaries and discussion board). If possible, report adherence to exercises in daily life between intervention sessions.
If appropriate, analyze how adherence is related to outcome.
Address the possible impact of adherence on intervention outcomes in the discussion section.
Strengths of this review are the systematic approach and application of independent coding by at least two of the authors, which was employed in every step of the review after initial title and abstract screening. This reduced the risk of selection bias. Our selection procedure led to the inclusion of studies on interventions that are comparable in core design characteristics (eg, multiple sequential sessions). In addition, a large number of publications could be included, reflecting the growth of the field in the past decade.
This review also has some limitations. Initial title and abstract screening was conducted by only one person. Although only publications that were clearly out of the scope of our review were excluded at this step, we cannot completely rule out the possibility that publications were excluded erroneously. Our review is further limited to studies evaluating interventions targeting common mental disorders (depression, anxiety, eating disorders, and substance-related disorders). Interventions targeting other disorders (eg, psychosis or bipolar disorders) were excluded because these conditions are not viewed as common mental disorders. Hence, our findings cannot be generalized across the whole field of electronic mental health research. Moreover, this review cannot draw any conclusions regarding the impact of adherence on outcome (eg, whether there are consistent findings in terms of strength or direction of associations), but examined whether a dose-response relationship was addressed at all in the individual publications. Secondary publications on adherence were not considered, as the CONSORT-HEALTH guidelines (Eysenbach, 2011 [
In summary, most publications included information on adherence and addressed adherence in the discussion. The most frequently reported usage metric was full intervention completion. There was substantial variety in the usage metrics utilized to operationalize adherence, which impedes comparisons regarding adherence between studies and interventions. Only one in three publications reported on adherence in the abstract. Publications often are screened by abstract and sometimes even evaluated only by the abstract. Results presented in the abstract are more likely to be disseminated by journalists than results presented elsewhere in a manuscript [
List of included studies.
electronic health
Consolidated Standards of Reporting Trials of Electronic and Mobile HEalth Applications and onLine TeleHealth
randomized controlled trial
The ICare Consortium has received funding from the European Union’s Horizon 2020 research and innovation program under grant agreement number 634757.
None declared.