This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Although Web-based interventions have been shown to be effective, they are not widely implemented in regular care. Nonadherence (ie, participants not following the intervention protocol) is an issue. By studying the way Web-based interventions are used and whether there are differences between adherers (ie, participants that started all 9 lessons) and nonadherers, more insight can be gained into the process of adherence.
The aims of this study were to (1) describe the characteristics of participants and investigate their relationship with adherence, (2) investigate the utilization of the different features of the intervention and possible differences between adherers and nonadherers, and (3) identify what use patterns emerge and whether there are differences between adherers and nonadherers.
Data were used from 206 participants that used the Web-based intervention Living to the full, a Web-based intervention for the prevention of depression employing both a fully automated and human-supported format. Demographic and baseline characteristics of participants were collected by using an online survey. Log data were collected within the Web-based intervention itself. Both quantitative and qualitative analyses were performed.
In all, 118 participants fully adhered to the intervention (ie, started all 9 lessons). Participants with an ethnicity other than Dutch were more often adherers (χ2
1=5.5,
By using log data combined with baseline characteristics of participants, we extracted valuable lessons for redesign of this intervention and the design of Web-based interventions in general. First, although characteristics of respondents can significantly predict adherence, their predictive value is small. Second, it is important to design Web-based interventions to foster adherence and usage of all features in an intervention.
Dutch Trial Register Number: NTR3007; http://www.trialregister.nl/trialreg/admin/rctview.asp?TC=3007 (Archived by WebCite at http://www.webcitation.org/6ILhI3rd8).
Depression has a high prevalence that poses a large burden on the health care system. Research shows that easily accessible interventions for indicated prevention (targeted at people at risk) are essential and can be cost-effective [
Although Web-based interventions have been shown to be effective, Web-based interventions are still not widely implemented in regular care [
In recent years, adherence has gained considerable attention. Eysenbach coined the phrase law of attrition [
There has been research into the usage and use patterns of Web-based interventions. Descriptive studies of freely accessible interventions have shown that they attract a considerable number of visitors, but that these visitors often interact with or access a fraction of what is possible in the intervention [
In addition to adherence as a process, there are still many questions regarding characteristics of respondents that may predict adherence. Studies have investigated the predictive value of demographics and disease-related measures (eg, [
This paper presents analyses of log data collected in a study into the adherence and effectiveness of a Web-based intervention for the prevention of depression, in which 118 of the 239 participants (49.4%) adhered to the intervention (ie, started all 9 lessons) [
The analyses described in this paper were performed on data collected in the parent study on the adherence and effectiveness of the Web-based intervention for the prevention of depression [
Following Van Gemert-Pijnen et al [
The Web-based intervention called Living to the Full is based on Acceptance and Commitment Therapy (ACT) [
The intervention was developed employing methods from the CeHRes Roadmap for eHealth development [
Personal home page of the Web-based intervention with the elements included for all participants.
For this study, the Web-based intervention was implemented in a research setting, namely at the University of Twente, the Netherlands. Participants could access the Web-based intervention at any time, from any place, free of charge. After finishing a lesson, participants could proceed to the next lesson after receiving feedback. This feedback was provided when a participant viewed all psycho-educational material and completed all exercises. Furthermore, feedback was sent at least 5 days after the participant started the lesson (see
Web-based interaction with the system consisted of doing online exercises, using multimedia content, and using personalized features. Interaction in the form of feedback messages (human or automated) was provided within the system as well. Furthermore, interaction with the system occurred through automated email messages that were sent to the participants’ email address to remind them to start, continue, or complete a lesson. For participants who signed up for short message service (SMS) coaching (see following paragraph), interaction also took place via their mobile phone. This interaction was 1-directional; there was no possibility to reply. Furthermore, all participants had the opportunity to contact the research staff by telephone, although this possibility was rarely used (approximately 5 phone calls during the intervention period in total).
Although the components of the intervention are not the focus of this study, this section will give a short overview of each of the levels of the components to be able to place the data presented in this study in its context. A detailed description can be found in
The source of support was either human or automated. To isolate the effect of the source of support, both conditions were designed as comparable as possible regarding length of feedback messages, tailored content, and presentation (including a photo of the counselor). To maintain the unique differences between human and automated support (increased possibility for interaction in human support and the increased possibility for timely feedback in automated support), participants in the human support condition had the opportunity to ask questions to their counselor, and participants in the automated support condition received 1 additional Web-based instant feedback message per lesson.
Participants in the condition that included SMS text messages had the opportunity to turn the SMS coach on. This SMS coach sent 3 predesigned text messages each week to a mobile phone number provided by the participant. The text messages were written by the researchers before the study started and the content was based on the results of the development study of the intervention [
The high experience condition contained additional multimedia and interactive material in the form of short movies, interactive exercises, and multimedia presentations of metaphors.
The intervention contained a success story for each lesson. For the high-tailored condition, each success story was tailored on 4 of the following aspects: gender, age, marital status, daily activity, most prominent symptom, and reason for participating in the Web-based intervention. The stories were tailored to a different combination of aspects each week and not on all aspects to maintain the credibility of the stories. In the low-tailored condition, a standard success story was presented each week.
The high-personalization condition included personalized content that was adapted (the system shows the motto and picture selected by the participant; the system shows the most important values selected by the participant) and adaptable (possibility to create a personal top 5 of aspects from the course that the participant found most important).
Characteristics of participants were collected at baseline by using an online questionnaire. Depressive symptoms were measured with the CES-D (20 items, score 0-60; higher=more depressive symptoms [
Usage of the Web-based intervention was measured objectively by log files. From these log files, adherence could be extracted. Adherence was defined as a participant starting lesson 9, because the intervention is intended to be used during the 9 lessons.
The log files contained a record of actions taken by each participant with for each action the following information: unique participant identification number, action type, action specification, and time and day. The action types that were logged were log-in, log-out, start lesson, start mindfulness, download mindfulness, view success story, view feedback message, start video, turn on SMS coach, turn off SMS coach, and view text message. Action specifications were, for example, the name of the mindfulness exercise started or which text message was viewed.
Descriptive analyses of use patterns were performed on 20 arbitrarily selected participants; 5 early nonadherers (ie, did not start lesson 5), 5 late nonadherers (ie, started lesson 5 but did not start lesson 9), and 10 adherers. We divided the nonadherers into early and late nonadherers to explore whether there were differences between these groups. It may be that people who nonadhere early have different reasons for nonadhering than late nonadherers. These early reasons may be more general aspects that become clear at an early stage (eg, the content is not attractive to them or the format of the intervention does not match their expectations). Late reasons may be more related to the process of the intervention or to the motivation (eg, it is hard to spend enough time each week on the intervention). It may be that late nonadherers are more similar to adherers and are easier to persuade to become adherers, whereas for early nonadherers, the intervention may simply not be suitable. Although the reasons for early or late nonadherence cannot be derived from this research, the results can show whether late nonadherers are more similar to adherers regarding their usage of the intervention. Effort was made to ensure that selected participants had the same distribution of demographic characteristics and randomized group as the full sample. Furthermore, we only selected participants who did not start to nonadhere in lessons 2, 5, or 8 because these were the lessons we investigated and we wanted to avoid including patterns of participants who did not complete the lessons under investigation. See
Statistical analyses were done using PASW 18 (Predictive Analytics Software; IBM, USA). Differences between adherers and nonadherers were investigated using 1-way analyses of variance (ANOVA) and chi-square tests (χ2). Logistic regression was used to assess whether baseline characteristics predicted adherence. Because of the exploratory nature of the logistic regression, all predictor variables were added at once, using the enter method.
Baseline demographics and outcome measures of the 206 participants who used the intervention are shown in
The average number of lessons started was 6.9 out of a possible 9, and 57% of the participants in this study completely adhered to the intervention (mode and median = 9 lessons).
To explore the possible predictive value of baseline characteristics for adherence (ie, starting all 9 lessons), we performed an exploratory logistic regression with all baseline characteristics showed in
Baseline demographics and outcome measures of all participants, and differences between adherers and nonadherers.
Participant characteristic | Total |
Adherers |
Nonadherers |
|
Age (years), mean (SD) | 44.7 (12.5) | 45.2 (12.6) | 43.9 (12.3) | .47 |
Gender (women), n (%) | 150 (72.8) | 92 (78.0) | 58 (65.9) | .05 |
|
|
|
|
.02 |
|
Dutch | 188 (91.3) | 103 (87.3) | 85 (96.6) |
|
Other | 18 (8.7) | 15 (12.7) | 3 (3.4) |
|
|
|
|
.51 |
|
High | 139 (67.5) | 82 (69.5) | 57 (64.8) |
|
Middle | 53 (25.7) | 30 (25.4) | 23 (26.1) |
|
Low | 14 (6.8) | 6 (5.1) | 8 (9.1) |
|
|
|
|
.46 |
|
Married | 72 (35.0) | 45 (38.1) | 27 (30.7) |
|
Divorced | 41 (19.9) | 20 (16.9) | 21 (23.9) |
|
Widowed | 4 (1.9) | 3 (2.5) | 1 (1.1) |
|
Unmarried | 89 (43.2) | 50 (42.4) | 39 (44.3) |
|
|
|
|
.15 |
|
Paid job | 131 (63.6) | 69 (58.5) | 62 (70.5) |
|
Student | 16 (7.8) | 9 (7.6) | 7 (8.0) |
|
No job | 59 (28.6) | 40 (33.9) | 19 (21.6) |
CES-D, mean (SD) | 24.9 (6.9) | 24.5 (7.3) | 25.4 (6.5) | .35 |
HADS-A , mean (SD) | 9.7 (2.6) | 9.4 (2.5) | 10.0 (2.6) | .13 |
Logistic regression baseline characteristics and adherence.
Included | Ba (SE) |
|
OR (95% CI) |
Constant | 0.56 (1.82) | .76 |
|
Age | –0.01 (0.02) | .65 | 0.99 (0.96-1.02) |
Gender | 0.70 (0.35) | .046 | 2.02 (1.01-4.04) |
Ethnicity | 1.29 (0.70) | .07 | 3.63 (0.92-14.26) |
Education | 0.30 (0.26) | .25 | 1.35 (0.81-2.24) |
Marital status | –0.09 (0.14) | .53 | 0.92 (0.70-1.20) |
Daily activities | 0.35 (0.19) | .08 | 1.41 (0.97-2.06) |
CES-D | –0.01 (0.02) | .71 | 0.99 (0.95-1.04) |
HADS-A | –0.12 (0.07) | .07 | 0.89 (0.78-1.01) |
Need for Cognition | 0.02 (0.01) | .02 | 1.02 (1.00-1.05) |
Need to Belong | –0.33 (0.27) | .22 | 0.72 (0.43-1.21) |
Internet usage | –0.16 (0.09) | .06 | 0.85 (0.72-1.01) |
Internet experience | –0.05 (0.11) | .64 | 0.95 (0.77-1.18) |
aB: unstandardized coefficient.
Graph of lessons completed against proportion of participants.
From the log files, the number of times each participant performed an action in the Web-based application was extracted (
To examine in more detail the way participants interacted with the system during the lessons, the use patterns of 20 participants (5 early nonadherers, 5 late nonadherers, and 10 adherers) on lesson 2 (all selected participants), lesson 5 (late nonadherers and adherers), and lesson 8 (adherers only) were investigated.
There are many sessions that involve only a log-in and a log-out action, with less than a minute in between.
Adherers start the later lessons with a very short first session.
Many feedback messages are not read the first session after they are available.
There are many log-in actions shortly after another action.
User actions of adherers and nonadherers.
User actions | Adherers |
Nonadherers |
Total |
|
|
|
|
|
|
|
Total | 40.2 (19.8) | 14.4 (13.6) | 29.1 (21.6) |
|
Per lesson | 4.5 (2.2) | 3.2 (1.5) | 3.9 (2.0) |
|
|
|
|
|
|
Total | 22.9 (17.6) | 6.1 (7.8) | 15.7 (16.5) |
|
Unique messages | 12.0 (5.2) | 3.8 (3.7) | 8.5 (6.1) |
|
Unique messages per lesson | 1.3 (0.6) | 0.8 (0.6) | 1.1 (0.6) |
|
|
|
|
|
|
Total started, mean (SD) | 7.8 (5.6) | 3.6 (3.3) | 6.0 (5.2) |
|
Unique started, mean (SD) %b | 3.6 (1.4) 72.0% | 2.1 (1.3) 74.3% | 2.9 (1.6) 73.0% |
|
Unique downloaded, mean (SD) %b | 2.6 (2.1) 51.5% | 1.1 (1.3) 37.7% | 1.9 (1.9) 45.6% |
|
Unique used, mean (SD) %b | 4.4 (1.0) 87.6% | 2.3 (1.3) 81.6% | 3.5 (1.5) 85.0% |
|
|
|
|
|
|
Total, mean (SD) | 8.8 (7.5) | 3.5 (3.5) | 6.5 (6.7) |
|
Unique, mean (SD) %b | 5.2 (2.8) 57.3% | 2.4 (1.9) 61.4% | 4.0 (2.8) 59.1% |
|
|
|
|
|
|
Participants that turned text message coaching on, nd | 19 | 7 | 26 |
|
Lessons turned on, mean (SD)e | 7.9 (2.6) | 2.4 (1.7) | 6.5 (3.4) |
|
Total messages viewed, mean (SD) | 14.3 (20.0) | 2.4 (3.7) | 9.6 (16.7) |
|
Unique messages viewed, mean (SD) %b | 8.4 (8.9) 31.0% | 1.8 (2.8) 14.9% | 5.8 (7.8) 24.6% |
|
|
|
|
|
|
Total, mean (SD) | 5.4 (6.1) | 2.0 (3.8) | 3.9 (5.5) |
|
Unique, mean (SD) %b | 3.5 (3.4) 38.6% | 1.3 (2.3) 25.5% | 2.5 (3.2) 32.9% |
aLog-ins within 30 minutes of the previous log-in were not counted to make the log-ins reflect the number of sessions more accurately; Log-ins per started lesson: number of log-ins divided by the number of the last lesson started.
b% = unique actions/possible actions. For adherers, the number of possible actions is the total number of available actions of that kind in the whole intervention. For nonadherers, the number of possible actions is the total number of available actions in all lessons that the participant started.
cOnly for participants in the condition that included text message coaching; n=105; adherers n=63; nonadherers n=42.
dThe number of participants that turned the text message coach on at least 1 time.
eThe number of lessons the text message coach was turned on for the participants that turned the text message coach on at least 1 time.
f Only for participants in the high experience condition; n=116; adherers n=65; nonadherers n=51.
Mean number of sessions and duration for early nonadherers (n=5), late nonadherers (n=5), and adherers (n=10).
Variable | Early nonadherers, |
Late nonadherers, |
Adherers, |
|||
Lesson | 2 | 2 | 5 | 2 | 5 | 8 |
Total sessions | 2.8 (1.6) | 4.4 (1.5) | 4.0 (1.6) | 5.5 (2.6) | 4.3 (1.3) | 4.0 (1.9) |
Sessions to complete lesson | 1.8 (0.8) | 2.0 (1.2) | 2.8 (1.6) | 3.5 (2.0) | 2.8 (0.9) | 1.9 (0.9) |
Total duration of session (min) | 36.2 (44.8) | 64.0 (45.2) | 38.8 (33.3) | 101.9 (55.6) | 125.6 (99.8) | 114.0 (110.4) |
Time in between sessions (days) | 6.7 (4.1) | 10.0 (4.1) | 10.8 (1.8) | 7.7 (1.7) | 10.8 (6.1) | 9.6 (5.2) |
The aims of this study were to (1) describe the characteristics of participants and investigate their relationship with adherence, (2) investigate the utilization of the different features of the intervention and possible differences between adherers and nonadherers, and (3) identify what use patterns emerge and whether there are differences between adherers and nonadherers.
The participants in this study were primarily Dutch females with a higher education level and a paid job. This group is similar to the group reached by many Web-based or eHealth interventions (eg, [
Overall, of the 206 participants that used the application, 118 participants adhered to the intervention. Although we included the percentage of adherers by using these numbers, it should be noted that we only report on participants that started lesson 1. The true adherence derived from all participants is 49.4% (118/239) [
Our results on the usage of the different features mirror the results of studies on the usage of freely available Web-based interventions in that participants do not use all the features that they can use [
Significant differences between the user actions of adherers and nonadherers (ie, adherers showed more log-ons per lesson, downloaded more mindfulness exercises, and viewed more text messages than nonadherers) indicate that adherers not only have more endurance regarding usage during the full duration of the intervention, but are also more engaged with the intervention compared to nonadherers. This mirrors the finding that adherers show more involvement with the intervention [
The average number of feedback messages viewed per lesson was below 1 for adherers and nonadherers, which shows that not all feedback messages have been viewed. Receiving feedback was the most wanted and expected feature of a Web-based intervention according to the participants in our development study, and providing support has been shown to have a positive effect on the effectiveness of Web-based interventions [
Our analyses of the use patterns of 20 participants over 3 different lessons, provided us with useful insights. This more qualitative analysis confirmed our quantitative results on user actions: adherers are overall more engaged, they use more sessions, and spend more time with the intervention. Moreover, the analyses of the use patterns show us that there may be a difference between early nonadherers, late nonadherers, and adherers, in which late nonadherers are more similar to adherers in the number of sessions, but have a shorter duration of sessions, which is more similar to early nonadherers. This seems to fit our hypothesis that there is a difference between early and late reasons for nonadherence. Late nonadherers may be more similar to adherers and they may be easier to persuade to become adherers, whereas the intervention may simply not be suitable for early nonadherers. This should be investigated in future research.
By identifying differences between adherers and nonadherers, it becomes possible to screen for these wrong patterns and identify participants that are at risk to become nonadherers. This provides the opportunity to intervene, for example, by notifying these participants that they have a use pattern that increases the likelihood for nonadherence and suggesting a more appropriate use pattern. This combination of monitoring and self-monitoring of behavior and providing suggestions for different behavior are thought to be persuasive strategies for behavior change [
Our in-depth analyses of the use patterns presented in
For this study, we used the log data of the Web-based intervention itself. This allowed us to identify actions of specific participants and relate them to whether the participant adhered to the intervention or not. Other studies have advocated the use of Google analytics, for example [
A limitation of this study is that we analyzed and interpreted log data without actively involving the participants. We did not ask participants why they used the intervention the way they used it. This information may have made it easier to interpret the data and to check whether our interpretation is correct. On the other hand, it is important to use objective log data and not to rely on subjective measures of how participants state that they used the intervention, because subjective data on usage are likely to be less accurate. Another limitation is the issue of generalizability. Our study used data from 1 intervention for the prevention of depression, which was used by primarily higher-educated Dutch women. Furthermore, we only investigated the use patterns of a small sample of these participants. The observed use patterns may be specific for this group using this intervention. However, many interventions, especially mental health interventions, have similar characteristics [
An interesting area for research can be found in a new way of analyzing the use patterns and investigating whether it is useful and feasible to intervene during the use of the intervention on the basis of the analyses of real-time use patterns. An earlier step might be to identify use patterns that are related to adherence and to design or redesign interventions in such a way to promote these use patterns. A different area of future research lies in the investigation of a more pragmatic way to identify participant characteristics that may influence or predict adherence, following the persuasion profiling approach [
In conclusion, we can say that using log data combined with baseline characteristics of participants of the intervention Living to the Full provided valuable lessons for redesign of this intervention and the design of Web-based interventions in general. First, although characteristics of respondents can significantly predict adherence, their predictive value is small. Therefore, we should look into other ways of classifying participants to make useful predictions about how individual difference may influence adherence. Second, it is important to design Web-based interventions to foster adherence and usage of all features in an intervention. A possibility for this is a smarter system that logs the current use pattern of a participant and intervenes when necessary, for example, by providing feedback or links to features that have not been accessed yet.
Description of parent study.
Detailed description of the five intervention factors.
Characteristics of respondents for analyses of usage patterns.
User actions, duration and time between sessions per participant per lesson.
CONSORT-EHEALTH checklist V1.6.2 [
Acceptance and Commitment Therapy
analysis of variance
Center of Epidemiological Studies Depression scale
Hospital Anxiety and Depression Scale
randomized controlled trial
short message service
The authors were involved in the development of the Web-based intervention Living to the Full.