This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Attrition is a noted feature of eHealth interventions and trials. In 2005, Eysenbach published a landmark paper calling for a “science of attrition,” suggesting that the 2 forms of attrition—nonusage attrition (low adherence to the intervention) and dropout attrition (poor retention to follow-up)—may be related and that this potential relationship deserved further study.
The aim of this paper was to use data from an online alcohol trial to explore Eysenbach’s hypothesis, and to answer 3 research questions: (1) Are adherence and retention related? If so, how, and under which circumstances? (2) Do adherence and retention have similar predictors? Can these predictors adequately explain any relationship between adherence and retention or are there additional, unmeasured predictors impacting on the relationship? (3) If there are additional unmeasured predictors impacting on the relationship, are there data to support Eysenbach’s hypothesis that these are related to overall levels of interest?
Secondary analysis of data from an online trial of an online intervention to reduce alcohol consumption among heavy drinkers. The 2 outcomes were adherence to the intervention measured by number of log-ins, and retention to the trial measured by provision of follow-up data at 3 months (the primary outcome point). Dependent variables were demographic and alcohol-related data collected at baseline. Predictors of adherence and retention were modeled using logistic regression models.
Data were available on 7932 participants. Adherence and retention were related in a complex fashion. Participants in the intervention group were more likely than those in the control group to log in more than once (42% vs 28%,
In this dataset, adherence and retention were related in a complex fashion. We propose a possible explanatory model for these data.
International Standard Randomized Controlled Trial Number (ISRCTN): 31070347; http://www.controlled-trials.com/ISRCTN31070347 (Archived by WebCite at http://www.webcitation.org/6IEmNnlCn).
In a landmark paper published in 2005, Eysenbach [
To identify papers responding to Eysenbach’s call, we undertook a search of PubMed for papers published between 2005 and the end of 2011 that addressed either form of attrition. The search strategy combined the concepts of Web-based interventions with attrition (either nonusage attrition/adherence or dropout attrition/retention). Of 2581 unique citations, more than 60 papers reported relevant information, including 5 systematic reviews [
Adherence to any specified intervention may be related to characteristics of the intervention, characteristics of the user, or characteristics of the condition addressed by the intervention. Characteristics of the intervention that may improve adherence to the intervention include a strong theoretical foundation [
Improving retention to studies has received less attention in the eHealth literature than improving adherence to interventions. For online questionnaires, the appearance, order, relevance, length, and origin of the questionnaire all seem important [
In Eysenbach’s original paper, he posited that the 2 forms of attrition were related to one another by a single underlying mechanism—losing interest—and called for empirical studies to test this hypothesis [
In a systematic review and meta-analysis of papers published before 2009, Cugelman et al [
In summary, although many researchers have provided data contributing to a science of attrition, there remain many unanswered questions:
Are adherence and retention related? If so, how, and under which circumstances?
Do adherence and retention have similar predictors? Can these predictors adequately explain any relationship between adherence and retention or are there additional, unmeasured predictors influencing the relationship?
If there are additional unmeasured predictors affecting the relationship, are there data to support Eysenbach’s hypothesis that these are related to overall levels of interest?
This paper addresses these questions through secondary analysis of an Internet-based trial of an online intervention to help heavy drinkers reduce their alcohol consumption.
Ethical approval was obtained from University College London ethics committee.
The data reported here were taken from the Down Your Drink (DYD) randomized controlled trial (DYD-RCT; ISRCTN: 31070347), a trial of an online intervention to help hazardous or harmful drinkers reduce their alcohol consumption [
The intervention website was a theoretically informed website with multiple interactive features. It contained 3 phases. Phase 1 (“It’s Up to You”) used the principles of motivational interviewing to enable the user to reach a considered decision on whether to change their alcohol consumption. Phase 2 (“Making the Change”) used cognitive behavioral therapy techniques to help users reduce their consumption, whereas Phase 3 (“Keeping on Track”) focused on relapse prevention, a further cognitive behavioral approach. Interactive e-tools, such as the drinking episode diary, provide opportunities for users to reflect on the role alcohol plays in their life and consider alternatives [
The primary outcome point was 3 months after randomization. Data collected at follow-up included past-week alcohol consumption (the primary outcome), the EQ-5D, and the same secondary outcome measure completed at baseline. Data were collected online, with participants sent an email request for follow-up data. The email contained an embedded hotlink that led through to the questionnaires. Up to 3 reminders were sent at 7-day intervals to nonresponders, with the final reminder asking participants to tell us their total past-week alcohol consumption only. This follow-up regime was selected on the basis of our pilot study, exploring different methods of optimizing retention [
For the purposes of this paper, the 2 outcomes of interest were adherence to the intervention and retention in the trial. Adherence to the intervention was categorized by number of log-ons to the site into 3 groups: users who logged in once only, users who logged in twice, and users who logged in 3 or more times. This categorization was empirically based because the content was highly person-centered so that there were no recommended patterns of use. Retention was treated as a binary variable: participants either did or did not provide follow-up data at 3 months.
The independent variables were the data collected at baseline. Demographic variables were age, gender, educational status (categorized as having vs not having a university degree), marital status (categorized as married/living with partner vs single), having children (categorized as 1 or more vs none), ethnicity (white British or anything else), country of residence (Britain vs rest of the world), and providing offline contact details (provided either address or phone number vs not providing any offline details). Clinical variables were past-week alcohol consumption in units (in which 1 unit is equivalent to 8 g of ethanol), EQ-5D scores, and scores on 2 single-item measures of confidence and intention, both scored from 1 to 5 with 5 indicating the highest level of confidence or intention. For these items, participants were asked, “How confident are you in your ability to reduce your drinking?” and “How strong would you rate your intention to reduce your drinking in the next 3 months?” The EQ-5D scores were obtained from the 5-item questionnaire following the standard procedures [
Predictors of adherence and retention were modeled by using logistic regression models with outcomes whether a user (1) logged in twice or more, (2) logged in 3 times or more, and (3) responded at 3 months. To explore the association between adherence and retention, dummy variables for exactly 2 log-ins and 3 or more log-ins were included in model 3.
To select a set of independent variables for all adjusted analyses, we first fitted models 1 to 3 using all the demographic and clinical baseline variables listed previously as independent variables. Past-week alcohol consumption was log-transformed after adding 1 unit/week. For each independent variable, we found the smallest
Data were available on 7932 participants. Demographic and alcohol-related characteristics at baseline are presented in
Of the 7932 participants, 5165 (65%) logged in once only, 1538 (19%) logged in twice, and 1229 (16%) logged in 3 or more times (
Retention also varied by arm. The overall response rate at 3 months was 45% (3528/7932). Participants in the intervention arm were less likely to respond than those in the control arm (40% vs 49%,
These data appeared to conflict with Eysenbach’s hypothesis of a common factor in adherence and retention because participants in the intervention arm were both more likely to log in to the intervention and less likely to respond to follow-up than those in the control group. However, within each arm, the data were supportive of the Eysenbach hypothesis, with participants who logged in more frequently being more likely to respond at 3 months (
These data suggest that the answer to our first research question (are adherence and retention related?) is yes. In this dataset, adherence and retention are related, but the relationship is not straightforward. Overall, participants who logged in more frequently were more likely to respond at 3 months, but those in the intervention arm were both more likely to log in more than once and less likely to respond than those in the control arm.
Our second research question was “Do adherence and retention have similar determinants? Can these determinants adequately explain any relationship between adherence and retention or are there additional unmeasured determinants impacting on the relationship?”
To address this question, we first explored baseline predictors of adherence and retention (
Where demographic characteristics were found to predict adherence, they tended to also predict retention and vice versa, although the relationships did not always reach statistical significance. Thus, being older, female, having a university degree, and not having children were all predictive of being more likely to log in more frequently and provide follow-up data at 3 months (
In contrast, the relationship between alcohol-related characteristics including past-week consumption, intention to reduce consumption, and level of confidence in the ability to reduce consumption, and adherence or retention went in opposite directions (
These data suggest that the answer to our second research question was that although there was some similarity between the determinants of adherence and the predictors of retention, the relationship between adherence and retention could not be wholly explained by the predictors measured at baseline, suggesting there were additional unmeasured confounders affecting this relationship.
Our third research question was whether the data supported Eysenbach’s hypothesis that participant’s overall level of interest was the factor responsible for any relationship between adherence and retention. As discussed previously, the data pertaining to this question were somewhat contradictory, with adherence and retention positively correlated within each arm of the trial but not across arms because participants in the intervention arm were more likely to adhere but less likely to respond than participants in the control arm.
Baseline characteristics of the study population (N=7932).
Characteristic | Intervention |
Control |
Age (years), mean (SD) | 37.97 (10.96) | 38.29 (10.78) |
Gender (female), n (%) | 2246 (57) | 2299 (58) |
Have university degree, n (%) | 2067 (52) | 2026 (51) |
White British, n (%) | 3317 (84) | 3316 (84) |
Have children, n (%) | 2052 (52) | 2027 (51) |
Provided offline address or telephone number, n (%) | 1559 (39) | 1528 (39) |
Past-week alcohol consumption in unitsa, mean (SD) | 57.68 (39.62) | 56.86 (38.09) |
EQ-5D, mean (SD) | 0.84 (0.19) | 0.84 (0.19) |
Confidenceb, mean (SD) | 2.77 (1.16) | 2.79 (1.15) |
Intentionb, mean (SD) | 3.82 (1.09) | 3.85 (1.06) |
a 1 unit = 8 g ethanol.
b Confidence and intention scored on a 5-point scale with 1 as the lowest and 5 as the highest.
Adherence and retention.
Number of log-ins |
Overall (N=7932) | Intervention (n=3970) | Control (n=3962) | ||||||
|
n (%) | Responded at 3 months (retention), n (%) | Adjusted OR (95% CI) | n (%) | Responded at 3 months (retention), n (%) | Adjusted OR (95% CI) | n (%) | Responded at 3 months (retention), n (%) | Adjusted OR (95% CI) |
1 | 5165 (65) | 2036 (39) | Ref | 2324 (59) | 792 (34) | Ref | 2841 (72) | 1244 (44) | Ref |
2 | 1538 (19) | 816 (53) | 1.79 |
745 (19) | 343 (46) | 1.79 |
793 (20) | 473 (60) | 1.88 |
3+ | 1229 (16) | 676 (55) | 1.92 |
901 (23) | 456 (51) | 2.12 |
328 (8) | 220 (67) | 2.58 |
a
Baseline predictors of adherence and retention.
Characteristic | Logged in twice |
|
Logged in ≥3 times |
|
Responded at 3 months |
|
|
|
|
|
|
|
|
|
|
|
Per 11-year increase | 1.23 (1.16-1.30) | <.001 | 1.41 (1.31-1.52) | <.001 | 1.36 (1.29-1.44) | <.001 |
|
|
|
|
|
|
|
|
|
Females vs males | 1.12 (1.02-1.23) | .02 | 1.12 (0.83-1.28) | .09 | 1.35 (1.22-1.47) | <.001 |
|
|
|
|
|
|
|
|
|
Degree vs no degree | 1.24 (1.13-1.37) | <.001 | 1.31 (1.15-1.50) | <.001 | 1.17 (1.07-1.29) | .001 |
|
|
|
|
|
|
|
|
|
White British vs other | 0.89 (0.78-1.01) | .08 | 0.85 (0.72-1.00) | .06 | 1.24 (1.09-1.41) | .001 |
|
|
|
|
|
|
|
|
|
No children vs children | 1.12 (1.01-1.25) | .04 | 1.15 (0.99-1.33) | .06 | 1.25 (1.12-1.39) | <.001 |
|
|
|
|
|
|
|
|
|
Yes vs no | 1.1 (0.91-1.11) | .88 | 0.91 (0.79-1.03) | .14 | 1.20 (1.09-1.32) | <.001 |
|
|
||||||
|
Per 0.78-unit increase in womena | 1.13 (1.05-1.21) |
|
1.15 (1.03-1.27) |
|
0.87 (0.81-0.93) |
|
|
Per 0.78-unit increase in mena | 0.99 (0.93-1.07) |
|
0.97 (0.88-1.07) |
|
0.91 (0.85-0.97) |
|
|
|
|
|
|
|
|
|
|
Per 0.19-unit increasea | 1.02 (0.97-1.07) | .40 | 0.99 (0.93-1.06) | .80 | 1.07 (1.02-1.12) | .01 |
|
|
|
|
|
|
|
|
|
Per 1.15-unit increasea | 0.92 (0.87-0.97) | .001 | 0.93 (0.87-1.0) | .04 | 1.06 (1.01-1.12) | .013 |
|
|
|
|
|
|
|
|
|
Per 1.08-unit increasea | 1.27 (1.21-1.34) | <.001 | 1.45 (1.35-1.56) | <.001 | 0.86 (0.82-0.90) | <.001 |
|
|
|
|
|
|
|
|
|
Intervention vs control | 1.84 (1.68-2.03) | <.001 | 3.43 (2.99-3.94) | <.001 | 0.70 (0.64-0.76) | <.001 |
a Continuous predictors expressed as per 1 SD change.
Participants in the intervention arm were more likely to use the intervention and less likely to respond to requests for follow-up data at 3 months than those in the control arm. This relationship is likely to be causal because these data were obtained in a RCT in which the only difference between groups was the allocated intervention. We can conclude, therefore, that allocation to the intervention rather than control led to lower levels of follow-up.
Within each trial arm, there was a strong association between logging in more and being more likely to provide follow-up data at 3 months. As expected, the relationship between demographic variables, adherence, and retention tended to be in the same direction, in which factors associated with greater adherence were also associated with greater retention. However, the variables pertaining to alcohol consumption (past-week alcohol consumption, intention to reduce consumption, and confidence in ability to reduce consumption) tended to impact on adherence and retention in opposite directions. Participants who may be more likely to benefit from the intervention (heavier drinkers, drinkers with greater intention to reduce, and those with less confidence in their ability to reduce) made greater use of both intervention and control websites (higher adherence) but were less likely to provide follow-up data.
These data provide some answers to 2 of the 3 research questions posed. In this study, adherence and retention were related. Although there was some similarity between the predictors of adherence and the predictors of retention, the relationship between adherence and retention could not be wholly explained by the predictors measured at baseline, suggesting there were additional unmeasured confounders affecting this relationship.
Our third research question was whether any additional factors influencing the relationship were related to overall levels of interest as postulated by Eysenbach. Our data cannot directly address this question, but 1 possible explanatory model is presented in
The data also suggest that there are user characteristics which influence adherence that act in different ways. We hypothesize that demographic factors are indicative of an unmeasured or latent variable, which we could describe as a “propensity to comply.” Participants with higher propensity to comply are more likely to use the intervention more often and to respond to requests for follow-up data. The propensity to comply may overwhelm the effect of being allocated to the intervention arm on retention. In contrast, alcohol-related factors, including past-week alcohol consumption, intention to reduce consumption, and confidence in one’s ability to reduce consumption, increase use of the intervention and reduce retention.
There are, of course, other possible interpretations of the data. One alternative interpretation is that participants allocated to the active intervention were more frustrated by the gap between their expectations of the intervention and their experience of using it. The active site promised a complete suite of tools to help users make a decision and then act on it, but the low number of log-ons clearly suggested that most users did not use it as planned. It is possible that this disappointment diminished willingness to have contact with the researchers, leading to lower retention. In contrast, the control website only offered straightforward information and, thus, produced less disappointment, leading to greater openness to contact with the researchers, particularly where there were unmet needs as represented by unsuccessful attempts at behavior change.
Our data build on the available literature in this field, and may help explain some of the contradictory results seen previously. Previous papers have looked for simple relationships between demographic factors, such as age, gender, or education, and either adherence or retention. Our results suggest that a more complex model is required which takes other factors into account. It is also likely that these relationships will vary according to the population and the behavior or condition studied.
There are many strengths to these data. They were derived from a large online RCT that had automated randomization ensuring complete concealment of allocation and automated data collection procedures that ensured that all data obtained were of adequate quality for analysis. The large sample size of nearly 8000 participants allowed for highly powered multivariate analyses. The main weaknesses are related to this being a secondary analysis. The initial trial was not designed to address the research questions posed in this paper. We had no a priori definition of adherence and used number of log-ins as the simplest measure of adherence because previous data have shown that number of log-ins and number of pages visited are highly correlated [
Possible model to explain relationship between adherence and retention.
The data presented here can only be thought of as hypothesis generating. Clearly, further studies specifically designed to test this model and related questions are needed before any firm conclusions can be drawn. However, if confirmed in other studies, these data have implications for both intervention and trial design. Because the biggest predictor of adherence was allocation to the intervention rather than the control arm, researchers need to focus on ensuring that Web-interventions are attractive to the user. The literature suggests that this can best be done by ensuring that the intervention is theoretically informed, has strong perceived personal relevance and effectiveness for users (eg, through tailoring and ensuring credibility), and multiple push factors, both automated (eg, email or short message service text prompts) or human (eg, facilitation or coaching). Previous authors have suggested that it may be possible to identify subgroups of the population who are most likely to adhere [
The implications for trial design are more challenging to elaborate. If our hypothesis that low retention was related to users in the intervention group feeling that their needs had been met were to be confirmed, this potentially has profound implications for the design of Web-based trials of Web-based interventions.
In an online RCT of a Web-based intervention to help hazardous and harmful drinkers reduce their alcohol consumption, adherence and retention were related in a complex manner. Some user characteristics, particularly demographic variables, had a positive impact on both adherence and retention, whereas behavioral and related variables increased adherence and reduced retention. We have proposed various possible hypotheses to guide further study.
Alcohol Use Disorders Identification Test (consumption questions)
Down Your Drink
odds ratio
randomized controlled trial
None declared.