Published on 14.11.12 in Vol 14, No 6 (2012): Nov-Dec
Persuasive System Design Does Matter: A Systematic Review of Adherence to Web-Based Interventions
Background: Although web-based interventions for promoting health and health-related behavior can be effective, poor adherence is a common issue that needs to be addressed. Technology as a means to communicate the content in web-based interventions has been neglected in research. Indeed, technology is often seen as a black-box, a mere tool that has no effect or value and serves only as a vehicle to deliver intervention content. In this paper we examine technology from a holistic perspective. We see it as a vital and inseparable aspect of web-based interventions to help explain and understand adherence.
Objective: This study aims to review the literature on web-based health interventions to investigate whether intervention characteristics and persuasive design affect adherence to a web-based intervention.
Methods: We conducted a systematic review of studies into web-based health interventions. Per intervention, intervention characteristics, persuasive technology elements and adherence were coded. We performed a multiple regression analysis to investigate whether these variables could predict adherence.
Results: We included 101 articles on 83 interventions. The typical web-based intervention is meant to be used once a week, is modular in set-up, is updated once a week, lasts for 10 weeks, includes interaction with the system and a counselor and peers on the web, includes some persuasive technology elements, and about 50% of the participants adhere to the intervention. Regarding persuasive technology, we see that primary task support elements are most commonly employed (mean 2.9 out of a possible 7.0). Dialogue support and social support are less commonly employed (mean 1.5 and 1.2 out of a possible 7.0, respectively). When comparing the interventions of the different health care areas, we find significant differences in intended usage (p = .004), setup (p < .001), updates (p < .001), frequency of interaction with a counselor (p < .001), the system (p = .003) and peers (p = .017), duration (F = 6.068, p = .004), adherence (F = 4.833, p = .010) and the number of primary task support elements (F = 5.631, p = .005). Our final regression model explained 55% of the variance in adherence. In this model, a RCT study as opposed to an observational study, increased interaction with a counselor, more frequent intended usage, more frequent updates and more extensive employment of dialogue support significantly predicted better adherence.
Conclusions: Using intervention characteristics and persuasive technology elements, a substantial amount of variance in adherence can be explained. Although there are differences between health care areas on intervention characteristics, health care area per se does not predict adherence. Rather, the differences in technology and interaction predict adherence. The results of this study can be used to make an informed decision about how to design a web-based intervention to which patients are more likely to adhere.
J Med Internet Res 2012;14(6):e152
- Systematic review;
- web-based interventions;
- persuasive technology;
- behavior change
Web-based interventions for promoting health and health-related behaviors are seen in many variations and health care areas. According to Barak et al.  a web-based intervention is:
...a primarily self-guided intervention program that is executed by means of a prescriptive online program operated through a website and used by consumers seeking health- and mental health–related assistance. The intervention program itself attempts to create positive change and or improve/enhance knowledge, awareness, and understanding via the provision of sound health-related material and use of interactive web-based components.
A web-based intervention can involve therapy that lasts for a predetermined, fixed period of time. However, it can also be a continuous program with no specific end date that supports self-management among patients with a chronic condition. It is made up of different, inseparable aspects which, according to Barak et al , are as follows: program content, multimedia choices, interactive online activities, and guidance and supportive feedback.
Evidence exists to support the effectiveness of web-based interventions. Research has shown these interventions to be effective in different areas of health care [- ]. However, many evaluations of eHealth interventions report either no positive effects at all or only limited ones [ - ]. One of the issues that is frequently addressed is the problem of non-adherence [ , - ], which refers to the fact that not all participants use or keep using the intervention in the desired way. Research suggests that non-optimal exposure to the intervention lessens the effect of these interventions [ , ]. Gaining an insight into the factors that influence adherence should therefore be one of the main focus areas in any research study into web-based interventions. In this context, it is important to stress the difference between the terms “adherence” or “non-usage attrition” and “dropout.” Dropout, or dropout attrition, refers to participants in a study who do not fulfill the research protocol (eg, filling out questionnaires). This is not a focus area of this study. Adherence, or non-usage attrition, refers to the extent to which individuals experience the content of an intervention [ , ]. This is the focus of our study.
When looking at literature about adherence to a therapeutic regimen [, ], adherence is seen as the extent to which the patient’s behavior matches the recommendations that have been agreed upon with the prescriber. The term is often seen as a reaction to the term “compliance,” which has a more coercive connotation. Consequently, in adherence, the patient plays an active role in achieving this behavior [ ]. At the same time, there is also a norm or recommendation from a prescriber, which the patient tries to match. This recommendation is missing from the definitions of both adherence and non-usage attrition [ , ]. In this study, we elaborate on the definition by introducing the concept of “intended usage.” Intended usage is the extent to which individuals should experience the content (of the intervention) to derive maximum benefit from the intervention, as defined or implied by its creators. This matches the norm or recommendation from the definition of adherence to a therapeutic regimen. By comparing the observed usage of an individual to the intended usage of a web-based intervention, we can establish whether or not this individual adheres to the intervention. In this context, adherence is a process that cannot be assessed solely by measuring usage at the beginning and end of the intervention. Rather, it has to be assessed throughout the entire process to establish whether or not an individual adheres to the intervention at each and every step of the way. Finally, by comparing the observed usage of each individual to the intended usage of the web-based intervention, the percentage of individuals that adhere to the intervention can be calculated. This results in a more objective measurement of adherence, which can then be compared to other interventions, even if the intended usage is different.
Adherence to web-based interventions has been the subject of research for some time. Many studies focus on whether and which respondents’ characteristics can explain variations in adherence [, , , ]. Although this is a very important line of study, it seems to take the technology of web-based interventions for granted. Technology as a means to communicate the content has been neglected in research. Indeed, this technology is often seen as a black box: a mere tool that has no effect or value and serves only as a vehicle for the delivery of intervention content. In line with a recent viewpoint paper, we propose to examine the technology from a holistic perspective and see it as a vital and inseparable aspect of the web-based intervention [ ]. This approach has been recommended in recent literature [ , , , ] and has been the key point in the field of persuasive technology [ ], where there are examples of studies on the persuasive capacities of technology to support web-based interventions in the health care domain [ - ].
Recently, two systematic reviews on the influence of intervention factors on adherence to web-based interventions were published [, ]. Although both reviews provide valuable insights, we feel that there are shortcomings that limit the applicability of these results for our objectives. First, with regard to adherence, the study of Brouwer [ ] takes exposure to interventions delivered via the internet as the outcome measure. Exposure is seen as the number of times the user or patient logged on, the time spent on site, page views, etc, but these are static measurements unrelated to the usage intended by these interventions. This gives limited insights into the process of usage and adherence, which makes it difficult to compare different interventions and specify how well certain interventions are doing. A review by Schubart [ ] fails to distinguish between dropout and adherence. This approach limits the applicability of the results because, in real-life implementation of web-based interventions, there is no research protocol to adhere to, only the intervention. The results of Schubart’s review [ ] cannot be generalized to these situations because we do not know whether engagement is due to the research or the intervention.
Furthermore, regarding the intervention factors, both studies use an ad hoc classification of these factors without a theoretical foundation, which makes it difficult to generalize and explain the results. We consider a web-based intervention as consisting of content, interaction, and technology. And, although these aspects are inseparable, they can be looked at in a structured manner. Both earlier reviews use a classification that, in our opinion, has substantial overlap in the goals to be achieved with these aspects. For example, in the review by Brouwer , a distinction is made between interactive behavior change strategies and interactive elements. It is stated that the goal of interactive elements is to “improve the attractiveness of the intervention or to provide the option for more information,” but this is not mutually exclusive with interactive behavior change strategies. For example, a quiz is seen as an interactive element, but in our opinion it can also be used as a means of receiving tailored feedback or as a way to self-monitor your knowledge or behavior. Allocating a quiz to one of these categories is therefore problematic. The categorization of intervention factors in the review by Schubart [ ] lacks depth and tries to encompass in one single categorization both modality (ie, the channel through which content is delivered; for example, email or telephone) and strategy (eg, feedback).
The current study attempts to overcome these shortcomings by employing a more objective and comparable measurement of adherence to web-based interventions and a classification of technology based on persuasive technology literature.
From the field of persuasive technology we learn that technology has the capacity to be persuasive through its role as a tool, a medium, and a creator of experiences . Fogg’s definition of persuasive technology limits this field to human-computer interaction and does not include computer-mediated communication (ie, including interaction with a person). However, we feel that it is unnecessary and undesirable to separate these two aspects of technology, particularly in the area of health care, because a web-based intervention is made up of different, inseparable aspects. We therefore propose a broader application of the term “persuasive technology” to include both human-computer interaction and computer-mediated communication. Accordingly, regarding the aspects of a web-based intervention, we propose a more pragmatic conceptual division between technology (ie, all the features of the web-based intervention, including multimedia and online activities) and interaction (ie, all interactions between the user or patient and the intervention, a counselor, or peers), which is slightly different from the aspects proposed by Barak. Following Fogg’s work, Oinas-Kukkonen introduces a framework to classify technology in its persuasive functions [ ]. This persuasive system design (PSD) model, which is used, for example, in a study by Lehto and colleagues [ ], classifies features of the technology as primary task support, dialogue support, social support, and credibility support. By applying this model to web-based interventions, we can systematically look at how persuasive system design categories are used and investigate their possible influence on adherence.
This study investigates whether intervention characteristics and persuasive design affect adherence to a web-based intervention. Web-based interventions are applied in various health care domains and intuitively it seems that there are differences between web-based interventions aimed at people with a chronic condition, at lifestyle change, or at mental health, because of the target group, involvement with a health care professional, and duration of the interventions. However, the underlying principles may well be the same. Therefore, from an intervention perspective, there is no absolute need to see these areas as being so different from each other that they cannot be compared. Consequently, it is interesting to see whether the preconceptions about the differences can be confirmed and whether there is added value for researchers and designers in one area to look at interventions from a different area.
Our systematic review aims to answer the following research questions: (1) What are the key characteristics of web-based interventions in terms of technology and interaction? (2) Are there any differences in intervention characteristics between web-based interventions aimed at chronic conditions, lifestyle, or mental health? (3) What percentage of participants adhere to web-based interventions? (4) Which characteristics of web-based interventions related to technology and interaction are linked to better adherence? These insights can help us understand and reduce the impact of non-adherence.
We conducted a comprehensive literature search using the following bibliographic databases: Web of Knowledge, EBSCOhost, PiCarta, SciVerse Scopus, and ScienceDirect. We used a combination of the constructs “web-based,” “intervention,” “adherence,” and “health.” For each construct, we used several keywords (see) to ensure a broad coverage of published studies in our review. Following this search strategy, we identified 14,264 articles published up to 2011 Oct 26 (see for the full flow diagram of article selection).
The review is limited to studies of web-based interventions in the health care domain. The criteria used for including a study were: (1) it involved a web-based intervention for promoting health through behavioral change; (2) the web-based intervention was intended to be visited and used on more than one occasion; (3) the research included an assessment of the effect of the intervention; (4) the study reported objective, quantifiable measurements of usage for the intervention; and (5) the study was published in either English or Dutch. Exclusion criteria were as follows: (1) dropout attrition and non-adherence were indistinguishable; (2) the intervention was aimed at care providers or relatives of the “patient;” (3) the description of the intervention did not include information about the applied persuasive features of the technology; and (4) the web-based intervention was not primarily intended to be used through a computer or laptop at the user’s or patient’s home. In addition, we only included peer-reviewed, published articles.
Study Selection and Data Collection
The study selection was done in three steps. First, the titles of all retrieved articles were screened for eligibility by two authors (SK and RK). Second, the abstracts of all initially relevant articles were screened for eligibility by the same authors. Finally, the full text of all remaining publications was checked for inclusion by two authors (SK and RK or SK and JvG). In cases where the suitability of a study came into question during one of the steps, it was included in the next step. Disagreements about including the full text publication were discussed until agreement was reached. To check whether any eligible publications had been overlooked during the initial search process, the reference lists of all systematic reviews that were identified in the original search were checked to find additional publications that met our inclusion criteria.
The characteristics of all of the interventions that were included were coded by two researchers (SK and RK) using a data extraction form based on a protocol for the systematic review of eHealth technologies . Where possible, data was extracted using the CONSORT-EHEALTH checklist [ ]. For the extraction, we relied on information that was available in the published literature. The basis of the data extraction was the intervention, not the study itself. This meant that for some interventions data from more than one article was used. Furthermore, when a study described more than one web-based intervention (eg, a comparison of two web-based interventions), all web-based interventions were coded separately.
The following characteristics were coded:
The name of the intervention was recorded. If the intervention had no name, the intervention was named after the first author of the primary article about the intervention.
Behavior or Condition
The targeted behavior or condition of each intervention was recorded. Furthermore, we recorded the area of health care targeted by the intervention (chronic condition, lifestyle, or mental health).
Studies and Study Design
For each intervention, the studies that were used to code the characteristics of the intervention were recorded. Furthermore, we also recorded whether these studies were randomized controlled trials (RCTs) or observational studies without randomized control groups.
Intended usage was defined as the extent to which the developers of the intervention felt that the intervention should be used to achieve the desired effect ( 5ix). When this information was not reported, it was inferred from the description of the intervention. For example, interventions requiring patients to monitor their behavior and receive feedback once a week to achieve the desired effect were coded as intended to be used once a week.
All reported information regarding the usage of the intervention (related to its intended usage) was collected, including the number of times the user or patient logged on and the number of modules completed ( 6aii).
A percentage of adherence was calculated to enable us to compare the different interventions. We did this by calculating the percentage of participants that adhered to the intervention. For example, when the intended use of an intervention was “complete 8 modules” and 60 out of 100 participants completed 8 modules, the adherence was 60%. For each intervention that was included, we calculated one overall adherence percentage. When more studies about the same intervention yielded different adherence percentages, we calculated the overall adherence percentage using a weighted average, based on the number of participants in each study. Furthermore, when the study included a waiting list and the respondents in this waiting list received access to the intervention at a later stage, the adherence was calculated based on usage data for all participants, including the waiting list group.
The frequency of content updates for the web-based intervention for a participant was recorded. This could be based on new information being uploaded for all participants or on a new lesson becoming available for a specific participant.
The duration of the intervention in weeks was recorded.
For each intervention, we created a record indicating whether the setup was modular (ie, content is delivered in a sequential order, whereby new content is made available when the user reaches a certain point) or free (ie, all the content of the intervention is available to the user from the start).
All information about the interaction with participants was recorded ( 5viii, 5x, and 5xi). This interaction could be with the system (eg, automatic email reminders or a web-based automated response to filling out an exercise), with a counselor (eg, through email, telephone, or face-to-face meetings), or with peers (eg, through a discussion board, chat group, or face-to-face group sessions).
We recorded when interaction with the system, counselor, or peers took place through a different modality than web-based (face-to-face meeting, telephone, or SMS). An exception was made when the study protocol included a face-to-face meeting or telephone intake. This was not coded as interaction through a different modality because it was not part of the actual intervention.
Persuasive Technology in the Intervention
The applied principles of persuasive technology within the interventions were coded according to the PSD framework of Oinas-Kukkonen and Harjumaa . We omitted system credibility support because of an observed lack of reporting of these principles in the studies that were included. The elements from the PSD framework on primary task, dialogue, and social support, with the definitions and the coding scheme we used, are presented in . The coding scheme is somewhat modified for the purpose of this study and to account for the computer-mediated communication included. However, when coding the persuasive technology elements, the technology was central, not the content of the interaction. Therefore, when computer-mediated communication was present, the content of this communication was not coded as persuasive technology. For example, when a feedback message from a care provider contained praise, this was not coded as dialogue support. When the technology provided a praising message after the user had successfully filled out a diary entry, then it was coded. For each intervention, the elements that were present were coded, irrespective of whether the designers of the intervention deliberately included these elements as persuasive technology elements. To check for differences in interpretation when coding the persuasive technology elements, 10 interventions were coded by 2 researchers (SK and LvG). The interrater reliability, measured by Cohen’s kappa, was 0.91.
All data on each intervention was entered in SPSS version 19.0 (IBM Corporation, Somers, NY, USA), and we treated each intervention as a separate case. Descriptive data of the combined data of all included interventions on all variables were calculated using SPSS. Differences in variables between health care areas were calculated using Fisher’s exact tests (because of the small expectation values) and one-way analyses of variance. To investigate whether the characteristics of the included interventions could predict the observed adherence, we performed a hierarchical multiple linear regression analysis, using a block-wise “enter” method. The first block was related to the context of the web-based intervention and included the health care area (coded as dummy variables) and the study design (RCT vs observational), which other researchers have proposed to influence adherence or the effect of web-based interventions [, , ]. The second block relates to our concept of interaction as one of the aspects of a web-based intervention and consists of the frequency of interaction with a counselor, the system, and peers, as well as the modality employed. The third and fourth blocks relate to our concept of technology in a web-based intervention, where the third block contains the intervention characteristics intended usage, setup, updates, and duration, and the last block contains the categories of persuasive system design. It is important to note that we chose to include the categories, and not the separate elements in the multiple regression, because (1) the results could be biased when some elements are hardly used and these elements are entered as predictors; (2) entering all 21 elements increases the chance of a type I error; and (3) the PSD model has grouped the elements on their key benefits (when the benefits of the specific elements in a category are similar, then looking at the specific elements could cause the overall influence of the category to be missed).
The search yielded 7345 unique titles. After title, abstract, and full-text screening, 101 articles on 83 interventions were included (). In total, 315 articles were excluded based on the full text. The most common reason for exclusion was related to usage data: the lack thereof (n = 84) or the presentation of inadequate (ie, subjective or not usable for calculating adherence) usage data (n = 78). Other studies were excluded based on the studied intervention: not aimed at health promotion by changing behavior (n = 40); not primarily meant to be used from a computer or laptop at the user’s home (n = 41); not intended to be visited and used on more than one occasion (n = 34); or not targeted at the patient (n = 3). Twenty-seven publications were excluded because the study design did not include an assessment of the effect of the intervention (eg, when they only presented qualitative data on the design of an intervention) or when the study design did not provide unique usage data (eg, a study about the long-term effects of an intervention). Seven publications were excluded because of the description of the intervention or study: in 4 publications no information could be gathered on the applied persuasive features of the technology from the description of the intervention and in 3 publications the data on the number of participants and their usage of the intervention was unclear. Finally, in the case of one citation, the full text could not be retrieved; this citation was therefore excluded.
Characteristics of the Studies that Were Included
The 83 interventions that were included are presented in. Overall, 19 interventions targeted a specific chronic condition (diabetes was targeted most often with 6 interventions). Sixteen interventions targeted a lifestyle behavior (weight management was targeted most often with 7 interventions). Smoking cessation was also often seen (5 interventions were targeted solely on smoking cessation and 1 intervention included smoking cessation as one of multiple targeted behaviors). Finally, mental health was targeted most often in the studies that were included. Of these 48 interventions, 12 focused on social phobia, although it should be noted that these interventions are only from two research groups that extensively studied their interventions. Depression, panic disorder, and anxiety were also targeted frequently in the interventions that we included (10, 8 and 7 interventions, respectively).
presents an overview of the variables of the interventions that were coded and their distribution over the different areas (chronic condition, lifestyle, and mental health). Overall, we can see that most interventions were meant to be used once a week, were set up in a modular way, were updated once a week, and lasted for approximately 16 weeks (median duration 10 weeks). Face-to-face, telephone, and SMS support, or a combination of these modes, were infrequently used, with 4 interventions combining face-to-face and telephone support (interventions 3, 10, 33, and 72) and 2 interventions combining telephone and SMS support (interventions 24 and 81). Seventy-six per cent of the interventions included interaction of the participant with a counselor, and a similar percentage (73%) included some form of interaction with the system. A little over half of the interventions (53%) included interaction with peers, with and without counselor interaction. The average percentage of participants who adhered to an intervention is 50.3% (min 1%; max 93%). The values of each of the variables for each included intervention can be found in .
Differences in Intervention Characteristics between Health Care Areas
When comparing the interventions of the different health care areas using Fisher’s exact tests, we find significant differences on intended usage (P = .004), setup (P< .001), updates (P < .001), frequency of interaction with a counselor (P < .001), the system (P = .003), and peers (P = .017). When looking at the standardized residuals (data not shown), we can see where these differences are manifested. We see that lifestyle interventions are more often intended to be used less than once a month than interventions in the other areas. We see that mental health interventions are less often free in terms of their setup than the other two areas. Lifestyle interventions are more often not updated or updated without a known frequency. Regarding interaction with a counselor, we see that lifestyle interventions more often do not employ this feature. Furthermore, we see that lifestyle interventions more frequently include interaction with the system less than once a week. Finally, on interaction with peers, chronic interventions more often have interaction for which the frequency is not specified. One-way analyses of variance show that there are differences in duration (F = 6.068, P = .004) and adherence (F = 4.833, P = .010). Bonferroni post hoc analyses show that the difference in duration is between lifestyle and mental health interventions (lifestyle interventions are longer), whereas the difference in adherence is between lifestyle and chronic condition interventions and between lifestyle and mental health interventions (lifestyle interventions have a lower adherence rate). In sum, lifestyle interventions are longer, the intended usage is less frequent, they have fewer updates, there is less interaction with the system and a counselor, and there is lower adherence than interventions aimed at chronic conditions and mental health. Mental health interventions are less often free in their setup and interventions aimed at a chronic condition include interaction with peers more often, for which the frequency is not specified.
When examining the persuasive technology elements that are presented in, we see that a mean of 5.6 (median 5) out of a possible 21 elements were used within a web-based intervention. Primary task support shows the highest mean (2.9 out of a possible 7; median 3), while social support shows the lowest mean (1.2 out of a possible 7; median 1). One-way analyses of variance show that there is a significant difference between the use of persuasive technology elements for primary task support (F = 5.631, P = .005). A Bonferroni post hoc analysis shows that this difference is between lifestyle and mental health interventions, where lifestyle interventions employ a higher mean of elements than mental health interventions. Furthermore, we can see that in primary task support, tunneling is used most often (n = 75; 90%), closely followed by tailoring (n = 73; 88%). Tunneling is used in all included mental health interventions, but only in 10 (63%) of lifestyle interventions (significant difference; P < .001). Reduction and self-monitoring are less often used in mental health interventions than in the other areas (significant difference reduction, P = .033; and self-monitoring, P < .001). This is most strikingly seen in self-monitoring, which is used in 94% of lifestyle interventions, as opposed to 12% in the mental health interventions. Overall, rehearsal and simulation are used least of all out of the primary task support elements. From the dialogue support elements, reminders are most often used (n = 61; 74%) across all areas. Suggestion is the second most frequently used element (n = 24; 29%), although this is used more often in web-based interventions targeted at chronic conditions than in mental health (P = .008). Praise was not used in any of the interventions and rewards were used only in 3 interventions. In social support, we see that social facilitation is most often used (n = 43; 52%), with a significant difference between interventions aimed at a chronic condition (n = 14; 74% including social facilitation) and at lifestyle (n = 5; 31%; P = .046). Furthermore, social learning and social comparison are used reasonably frequently (respectively n = 31; 39% and n = 14; 17%), with mental health interventions predominantly contributing to these numbers (with a significant difference only for social learning: P = .044). Cooperation, on the other hand, is used in 2 lifestyle interventions and 1 chronic intervention, but in none of the mental health interventions (significant difference; P = .041). The other elements (normative influence, competition, and recognition) are hardly used. In sum, primary task support is most extensively employed while dialogue support and social support are sparsely employed. Tunneling, tailoring (primary task support), reminders (dialogue support), and social facilitation (social support support) are the most frequently used elements. On average, lifestyle interventions employ more primary task support elements than mental health interventions.
Predictors of Adherence
We performed a hierarchical multiple linear regression, using a block-wise “enter” method, to explore the predictors of adherence. Variables expected to predict adherence were entered in the analysis in blocks of related constructs, as specified in the methods section. The final model explained 55% of the variance in adherence. In this model, interventions studied with a RCT design (instead of an observational study), increased interaction with a counselor, more frequent intended usage, more frequent updates and more extensive employment of dialogue support significantly predicted better adherence.
In this systematic review, we have attempted to synthesize the combined knowledge of eHealth researchers to gain insights into the factors that affect adherence to web-based interventions in the areas of chronic conditions, lifestyle, and mental health. In this study, we viewed technology from a theoretical perspective and conceived adherence as an objective measurement that allows for comparison between different interventions.
We included 101 publications describing research into 83 interventions. Mental health interventions (n = 48) constituted the largest part of these interventions. Looking at the key characteristics of web-based interventions in terms of technology and interaction, it appears that the typical web-based intervention is meant to be used once a week, is modular in setup, is updated once a week, lasts for 10 weeks, includes interaction with the system, a counselor, and peers on the web, includes some persuasive technology elements, and results in about 50% of the participants adhering to the intervention.
However, to answer our second research question, there do appear to be differences between health care areas. Overall, lifestyle interventions are longer and less strict (more employ a free setup, less frequent intended usage, fewer updates, and less interaction) than interventions aimed at chronic conditions and mental health, which seems to result in lower adherence with lifestyle interventions. Mental health interventions follow the weekly, modular format the most, with only one intervention using a free setup. This may be explained by the difference in scope of lifestyle and mental health interventions: lifestyle interventions may be more oriented towards long-term changes, while mental health interventions are often aimed at treatment that is delivered in a short, strict format. However, interventions for a chronic condition are also aimed at a long-term change or goal, but these interventions are on average more strict than lifestyle interventions. More counselor involvement is likely to be an explanation because these interventions are often offered in a health care setting and we saw a significant difference between these areas.
Regarding persuasive technology, we see that primary task support elements are most commonly employed, especially in interventions aimed at chronic conditions and lifestyle. Tunneling, which is a technological result of a modular setup, is employed most often in mental health interventions and less frequently in lifestyle interventions. This difference is a logical result of the differences in setup between interventions in these areas. This finding is not surprising, taking into account that most mental health interventions are based on regular face-to-face therapy where psycho-education and behavior modification is usually delivered step-wise (see ). Tailoring, which is widely recognized as an important feature of effective health communication [ , ], is used in one form or another in 88% of the interventions. Strikingly, rehearsal, which is also seen as very important in learning and behavior change [ , ], is seldom employed. It may be that rehearsal is seen by the authors of the articles reviewed as such an obvious part of an intervention that a description of this process is omitted from the description of the interventions. If not, this should be a point of particular interest when (re)designing web-based interventions.
Only a mean of 1.5 out of a possible 7 dialogue support elements are employed per web-based intervention. It should be noted that we have not coded the elements that may be present in email-like messages sent by a counselor because we feel that this is part of the counselor interaction and not so much a part of the dialogue support that Oinas-Kukkonen  and Fogg [ ] describe. Reminders are the most frequently employed element. Studies have shown the importance of reminders in increasing adherence and in increasing the effectiveness of web-based interventions [ , ]. Therefore, we found it striking that 26% of the interventions did not include reminders in some way. Suggestion was the second most frequently used element and was employed more in interventions aimed at chronic conditions than mental health. This seems likely to be due to the focus of the interventions for chronic conditions being on coping with a condition and giving suggestions or strategies to achieve this, whereas in mental health interventions the focus is often more curative to “solve” a certain problem. Praise and rewards are seldom used, which may be a shortcoming when looking at the recent literature into serious gaming and gamification, where employing game-like strategies, such as praise and rewards, are expected to have positive effects on the outcomes of health interventions [ , ].
Social support is widely recognized as an important strategy in behavior change [, ] and it might be disappointing to see that, on average, only 1.2 out of a possible 7 elements are used per web-based intervention. Social facilitation was used in more than half of the interventions. It must be noted that here social facilitation means providing the opportunity to contact others using the same intervention; it does not say anything about whether the opportunity is actually used. In practical terms, this means that when an intervention includes a discussion board, social facilitation is employed, even when there are no posts on the discussion board. Social learning and social comparison were employed through, for example, obligatory posts of exercise answers on a discussion board or by providing a story by a user (real or fictive), including how he or she dealt with the situation. Cooperation, competition, normative influence, and recognition are seldom used and therefore provide areas in which web-based interventions might be improved. However, in this study, social support did not affect adherence, so more research is needed to investigate whether or not this area provides added value.
Our third research question was about the percentage of participants that adhere to web-based interventions. We found an average adherence of 50%, which confirms that non-adherence is an issue in web-based interventions. There was a wide range in the level of adherence, with 6 interventions scoring below 10% adherence and 5 interventions scoring 90% adherence or higher. Our last research question was aimed at determining which characteristics of web-based interventions relating to technology and interaction are related to better adherence. Using a hierarchical multiple linear regression, our final model explains 55% of the variance in adherence, which, in our view, is a substantial amount that provides valuable insights into the issue of adherence.
Interestingly, the first two models (including the context of the intervention and the interaction within the intervention) were not significant. It was only when aspects relating to the format of the intervention and the technology employed were entered that the model reached significance. In the final model, an RCT, as opposed to an observational study, significantly predicted better adherence. A likely explanation is that the observational studies in our review were mainly small pilot studies and large real-life studies. Pilot studies are likely to show lower adherence rates because the interventions are not fully tested and are improved after the outcomes of the pilot are known. Real-life observational studies have been shown to have lower adherence rates, which suggests that the formal structure of a trial is important for participants to adhere to an intervention . Furthermore, the selection processes of many RCTs make it likely that there is a difference in the participants in both settings, which contributes to the difference in adherence.
The frequency of interaction with a counselor was a significant predictor of adherence. This finding concurs with reviews of Brouwer , Schubart [ ], and other studies (for an overview see [ ]) that conclude that counselor or clinician support is related to greater exposure and engagement. Of the significant predictors in our study, this variable contributes the least. In our review, we have found no evidence that the frequency of interaction with peers is related to adherence. This is somewhat contrary to the results of Brouwer [ ], who concluded that peer support was related to greater exposure. In that study, exposure was seen as the time visitors spend on the website, which is very different from our definition of adherence. Furthermore, in this study, we coded the frequency of interaction, not merely whether there was any interaction or not. This resulted in 29% of interventions being coded as, “Yes, there is interaction with peers, but the frequency is unknown.” This frequency may vary to a large degree between these interventions, but without clear information we cannot make a distinction, which may have influenced our results.
In the final model, the frequency of interaction with the system seems to negatively influence adherence, although not significantly. This surprising finding may be explained by the fact that more interaction with the system meant, in many cases, that there was no interaction with a counselor. More frequent intended usage also predicts better adherence. This might seem counterintuitive, but might also mean that when people are expected to be more active they become more engaged with the system. Moreover, more frequent intended usage will, in many cases, lead to more frequent reminders and we know that reminders can positively influence adherence . That the provision of frequent updates is important was also seen in the review of Brouwer [ ] and is confirmed in this study.
Finally, more extensive employment of dialogue support is related to better adherence. This outcome was predicted by the persuasive system design model , but this study is, to our knowledge, the first to confirm this outcome related to adherence in a health setting. When looking at the other persuasive technology categories, we see that social support shows a trend towards a significant contribution to better adherence. We feel that this trend warrants further investigation. It might be that it has no significant predictive value in this study because of the limited use of social support elements in the included interventions. Interestingly, primary task support does not show any predictive value for adherence. This may well be explained by the purpose of the employment of primary task support. As indicated in the name, these elements make the primary task (ie, the goal of the intervention) easier, and are not so much focused on the process (ie, using the intervention or adhering to the intervention). It seems likely that these elements play a more important role in the effect of the intervention than in the adherence.
A final comment on the model for the prediction of adherence is on the different health care areas. We see that in the first model, lifestyle interventions, as opposed to mental health interventions, predict a lower adherence, but when adding the characteristics of the interventions in the model, this predictive value is negated. It seems that the health care area per se does not predict adherence, but the differences in the characteristics of the interventions in these areas do predict adherence.
Implications and Recommendations
Taking into account the results of this study, it seems reasonable to not only hope for adherence, but to plan for adherence when designing web-based interventions. Although 33 studies that are included in this review state that they have planned for adherence, it is remarkable that 18 state that encouraging adherence is a task for the counselor [- ] and one study included monetary incentives to promote adherence [ ]. Of the 15 studies that mention adapting the design of the intervention to increase adherence, 8 studies do so without any theoretical basis or reference [ - ], 4 studies make the adaptation the focus of their study [ - ], and 2 studies have adapted the design based on a prior study on the same intervention [ , ]. Overall, it seems that adapting web-based interventions to promote adherence is done in an ad-hoc manner and that a framework to guide researchers and developers in this area is needed. The PSD model [ ] may provide such a framework for the design of web-based interventions.
Moreover, it seems valuable to look much further than the health care area for which the intervention is being designed. Although each health care area has its own demands and limitations, the different areas might learn from each other’s strong points. Lifestyle interventions, although aimed at long-term goals, might benefit from incorporating segments with a more strict format and shorter duration. Mental health interventions might be extended to aim at more long-term goals like relapse prevention. They may therefore employ a less strict format, while being aware that adherence might become a larger problem. Moreover, mental health interventions might include the primary task support elements used in chronic condition and lifestyle interventions.
Furthermore, we now have evidence that certain intervention characteristics and persuasive technology can improve adherence. It seems that expecting a certain amount of engagement from the target group can actually be helpful in promoting adherence and is something that seems to be easy to implement in new and existing web-based interventions. We must keep in mind that the effect of intended usage might also be due to a bias among the participants when only those participants who agree in advance with a high level of engagement participate in such interventions. Duration seems harder to change. Cutting an intervention into shorter segments may be enough to improve adherence, but this should be investigated further. Including and possibly increasing the frequency of interaction with a counselor seems a more costly way to improve adherence and might, therefore, be a less than optimal starting point when specifically used as a strategy to increase adherence. Increasing dialogue support using persuasive technology seems to be a more cost-effective vantage point in this respect and may even be enhanced by the increasing use of mobile technology, which seems likely to, in turn, offer a valuable platform for introducing on-the-spot reminders and feedback.
Additionally, our results can be of value for blended care (ie, a combination of online and face-to-face care) by clarifying the crucial aspects for promoting adherence in web-based interventions. When it is not possible to adapt a web-based intervention to promote adherence, it may be feasible to include a face-to-face segment in the overall intervention at a crucial stage to make up for the predicted loss of adherence.
The results of this study can be used to make an informed decision about how to design a web-based intervention that has a greater likelihood of patient adherence. It must be noted, however, that we do not advocate a so-called “technology push” where technology is introduced only for the sake of the technology and the ability to create the technology. It should always be created in close collaboration with the target audience and with a clear goal to create a viable eHealth technology . This study provides insights into the choices one can make with the target audience.
In this study, we defined adherence as being the proportion of participants who use the intervention as it is intended to be used. By doing this, we have created an adherence measurement from objective data that is comparable between interventions. We feel that the study shows that this is a promising approach and this adherence measurement can be used for a wide variety of studies. However, to date, few studies report adherence as the measurement we have chosen to use. For review studies, this means that researchers have to define the intended use, search for the usage data that corresponded to this intended use, and then calculate the adherence. This might lead to a different interpretation of the usage data than the original authors intended. However, from our experience, we can say that as long as there is enough information on the intervention and the usage, it is feasible to calculate an objective and comparable adherence measurement. For intervention studies, we would advise researchers to at least provide the information needed (ie, intended usage and usage data related to this intended usage) to calculate this adherence measurement and, preferably, to state the calculated adherence percentage for easy comparison between interventions.
In this study, we have excluded many interventions because data about usage was absent or the usage data that was presented had no direct relationship to the intended use. For example, we excluded studies that only presented mean login data per week for all respondents and had an intended usage of once a week because these data do not show us which percentage of respondents logged in each week. This strict selection based on usage data might have introduced a bias in our included studies.
We have coded the web-based interventions included in this study based on the descriptions in the published literature. Although we have made an effort to find all the information in the published literature about each intervention, our coding was limited by the description of the interventions on paper. As is noted by other authors, the description of these interventions is varied [, , ], which makes it difficult to capture all the characteristics of each intervention, and this might have influenced our results. Initiatives to standardize and improve the description of web-based interventions like the consort statement for eHealth [ ], a protocol for systematic reviews in eHealth [ ], and guidelines for executing and reporting internet intervention research [ ] are therefore very necessary and will hopefully improve the possibility to compare eHealth technologies and learn from each other.
Lastly, a limitation of this review might be that we have only focused on the published literature. We have not included grey literature and have therefore included little real-life adherence data. As noted by Christensen , there is a difference between the usage of web-based interventions in a research setting and in a more real-life setting. We have tried to cope with this by using a strict definition of adherence, separating it from following the research protocol and filling out questionnaires, and by coding all interaction that might be the result of being part of a study as part of the intervention. Nonetheless, the limited amount of real-life data in our review might have influenced the results.
Overall, our results confirm the conclusions of prior studies [, ] that interaction with a counselor and regular updates promote adherence. Furthermore, the results of this review elaborate on the role of intervention characteristics (duration, setup, and intended usage) and persuasive technology, especially elements to support the dialogue. Finally, this study has provided practical recommendations to increase adherence when (re)designing a web-based intervention.
The data and results from this study provide numerous points of departure for future research. To increase our understanding of the characteristics of web-based interventions and their effect on adherence, it would be interesting to compare interventions that show high adherence with interventions that show low adherence using in-depth, qualitative analyses. The positive deviance approach used by Schubart  seems appropriate for this goal. Furthermore, it is interesting to test our statistical adherence model in experimental studies. Additionally, expanding the model by including the characteristics of participants seems to be relevant. Finally, exploring the relationship between persuasive technology, especially primary task support, and (clinical) outcomes of an intervention is likely to be a worthwhile line of research.
Conflicts of Interest
Multimedia Appendix 1
Keywords literature search.PDF File (Adobe PDF File), 29KB
Multimedia Appendix 2
Included interventions, targeted behavior or conditions, and studies.PDF File (Adobe PDF File), 112KB
Multimedia Appendix 3
Characteristics of, and adherence to, web-based interventions included in this study.PDF File (Adobe PDF File), 491KB
- Barak A, Klein B, Proudfoot JG. Defining internet-supported therapeutic interventions. Ann Behav Med 2009 Aug;38(1):4-17. [CrossRef] [Medline]
- Barak A, Hen L, Boniel-Nissim M, Shapira N. A Comprehensive Review and a Meta-Analysis of the Effectiveness of Internet-Based Psychotherapeutic Interventions. Journal of Technology in Human Services 2008;26(2/4):109-160. [CrossRef]
- Cuijpers P, van Straten A, Andersson G. Internet-administered cognitive behavior therapy for health problems: a systematic review. J Behav Med 2008 Apr;31(2):169-177. [CrossRef] [Medline]
- Spek V, Cuijpers P, Nyklícek I, Riper H, Keyzer J, Pop V. Internet-based cognitive behaviour therapy for symptoms of depression and anxiety: a meta-analysis. Psychol Med 2007 Mar;37(3):319-328. [CrossRef] [Medline]
- Wantland DJ, Portillo CJ, Holzemer WL, Slaughter R, McGhee EM. The effectiveness of Web-based vs. non-Web-based interventions: a meta-analysis of behavioral change outcomes. J Med Internet Res 2004 Nov 10;6(4):e40 [FREE Full text] [CrossRef] [Medline]
- Vandelanotte C, Spathonis KM, Eakin EG, Owen N. Website-delivered physical activity interventions a review of the literature. Am J Prev Med 2007 Jul;33(1):54-64. [CrossRef] [Medline]
- Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res 2010;12(1):e4 [FREE Full text] [CrossRef] [Medline]
- Norman GJ, Zabinski MF, Adams MA, Rosenberg DE, Yaroch AL, Atienza AA. A review of eHealth interventions for physical activity and dietary behavior change. Am J Prev Med 2007 Oct;33(4):336-345. [CrossRef] [Medline]
- Neve M, Morgan PJ, Jones PR, Collins CE. Effectiveness of web-based interventions in achieving weight loss and weight loss maintenance in overweight and obese adults: a systematic review with meta-analysis. Obes Rev 2010 Apr;11(4):306-321. [CrossRef] [Medline]
- Black AD, Car J, Pagliari C, Anandan C, Cresswell K, Bokun T, et al. The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Med 2011;8(1):e1000387 [FREE Full text] [CrossRef] [Medline]
- Kelders SM, Van Gemert-Pijnen JE, Werkman A, Nijland N, Seydel ER. Effectiveness of a Web-based intervention aimed at healthy dietary and physical activity behavior: a randomized controlled trial about users and usage. J Med Internet Res 2011;13(2):e32 [FREE Full text] [CrossRef] [Medline]
- van Gemert-Pijnen JE, Nijland N, van Limburg M, Ossebaard HC, Kelders SM, Eysenbach G, et al. A holistic framework to improve the uptake and impact of eHealth technologies. J Med Internet Res 2011;13(4):e111 [FREE Full text] [CrossRef] [Medline]
- Christensen H, Griffiths KM, Farrer L. Adherence in internet interventions for anxiety and depression. J Med Internet Res 2009;11(2):e13 [FREE Full text] [CrossRef] [Medline]
- Cugelman B, Thelwall M, Dawes P. Online interventions for social marketing health behavior change campaigns: a meta-analysis of psychological architectures and adherence factors. J Med Internet Res 2011;13(1):e17 [FREE Full text] [CrossRef] [Medline]
- Eysenbach G. The law of attrition. J Med Internet Res 2005;7(1):e11 [FREE Full text] [CrossRef] [Medline]
- Neil AL, Batterham P, Christensen H, Bennett K, Griffiths KM. Predictors of adherence by adolescents to a cognitive behavior therapy website in school and community-based settings. J Med Internet Res 2009;11(1):e6 [FREE Full text] [CrossRef] [Medline]
- Wangberg SC, Bergmo TS, Johnsen JA. Adherence in Internet-based interventions. Patient Prefer Adherence 2008;2:57-65 [FREE Full text] [Medline]
- Donkin L, Christensen H, Naismith SL, Neal B, Hickie IB, Glozier N. A systematic review of the impact of adherence on the effectiveness of e-therapies. J Med Internet Res 2011;13(3):e52 [FREE Full text] [CrossRef] [Medline]
- Manwaring JL, Bryson SW, Goldschmidt AB, Winzelberg AJ, Luce KH, Cunning D, et al. Do adherence variables predict outcome in an online program for the prevention of eating disorders? J Consult Clin Psychol 2008 Apr;76(2):341-346. [CrossRef] [Medline]
- Barofsky I. Compliance, adherence and the therapeutic alliance: steps in the development of self-care. Soc Sci Med 1978 Sep;12(5A):369-376. [Medline]
- Aronson JK. Compliance, concordance, adherence. Br J Clin Pharmacol 2007 Apr;63(4):383-384. [CrossRef] [Medline]
- Neve MJ, Collins CE, Morgan PJ. Dropout, nonusage attrition, and pretreatment predictors of nonusage attrition in a commercial Web-based weight loss program. J Med Internet Res 2010;12(4):e69 [FREE Full text] [CrossRef] [Medline]
- Danaher BG, Seeley JR. Methodological issues in research on web-based behavioral interventions. Ann Behav Med 2009 Aug;38(1):28-39. [CrossRef] [Medline]
- Fogg BJ. Persuasive Technology: Using Computers to Change What We Think and Do. San Francisco: Morgan Kaufmann; 2003.
- Chen F, Hekler E, Hu J, Li S, Zhao C. Designing for context-aware health self-monitoring, feedback, and engagement. 2011 Presented at: ACM Conference on Computer Supportive Cooperative Work; March 2011; Hangzhou, China p. 19-23. [CrossRef]
- Halko S, Kientz J. Personality and Persuasive Technology: An Exploratory Study on Health-Promoting Mobile Applications. In: Proceedings of the 5th International Conference on Persuasive Technology. 2010 Presented at: 5th International Conference on Persuasive Technology; June 2010; Copenhagen, Denmark p. 7-10. [CrossRef]
- Purpura S, Schwanda V, Williams K, Stubler W, Sengers P. Fit4life: the design of a persuasive technology promoting healthy behavior and ideal weight. In: Proceedings of the ACM CHI Conference on Human Factors in Computing Systems. 2011 Presented at: ACM CHI Conference on Human Factors in Computing Systems; May 2011; Vancouver p. 7-12. [CrossRef]
- Toscos T, Faber A, An S, Gandhi MP. Chick clique: persuasive technology to motivate teenage girls to exercise. In: Proceedings of the ACM CHI Conference on Human Factors in Computing Systems. Quebec, Canada; 2006 Presented at: ACM CHI Conference on Human Factors in Computing Systems; Aprill 2006; Montreal p. 24-27. [CrossRef]
- Brouwer W, Kroeze W, Crutzen R, de Nooijer J, de Vries NK, Brug J, et al. Which intervention characteristics are related to more exposure to internet-delivered healthy lifestyle promotion interventions? A systematic review. J Med Internet Res 2011;13(1):e2 [FREE Full text] [CrossRef] [Medline]
- Schubart JR, Stuckey HL, Ganeshamoorthy A, Sciamanna CN. Chronic health conditions and internet behavioral interventions: a review of factors to enhance user engagement. Comput Inform Nurs 2011 Feb;29(2 Suppl):TC9-T20. [CrossRef] [Medline]
- Oinas-Kukkonen H, Harjumaa M. Persuasive Systems Design: Key Issues, Process Model, and System Features. Commun Assoc Inf Syst 2009;24(1):28.
- Lehto T, Oinas-Kukkonen H. Persuasive features in web-based alcohol and smoking interventions: a systematic review of the literature. J Med Internet Res 2011;13(3):e46 [FREE Full text] [CrossRef] [Medline]
- Wentzel MJ, Karreman J, Van Gemert-Pijnen JE. Tutorial: data extraction for systematic reviews. Forthcoming 2013.
- Christensen H, Griffiths KM, Korten AE, Brittliffe K, Groves C. A comparison of changes in anxiety and depression symptoms of spontaneous users and trial participants of a cognitive behavior therapy website. J Med Internet Res 2004 Dec 22;6(4):e46 [FREE Full text] [CrossRef] [Medline]
- Kreuter M. Tailoring health messages: customizing communication with computer technology. Mahwah, N.J: Lawrence Erlbaum Associates; 2000.
- Noar SM, Benac CN, Harris MS. Does tailoring matter? Meta-analytic review of tailored print health behavior change interventions. Psychol Bull 2007 Jul;133(4):673-693. [CrossRef] [Medline]
- Abraham C, Michie S. A taxonomy of behavior change techniques used in interventions. Health Psychol 2008 May;27(3):379-387. [CrossRef] [Medline]
- Elder JP, Ayala GX, Harris S. Theories and intervention approaches to health-behavior change in primary care. Am J Prev Med 1999 Nov;17(4):275-284. [Medline]
- Fry JP, Neff RA. Periodic prompts and reminders in health promotion and health behavior interventions: systematic review. J Med Internet Res 2009;11(2):e16 [FREE Full text] [CrossRef] [Medline]
- Deterding S, Dixon D, Khaled R. Toward a Definition. 2011 Presented at: ACM CHI Gamification Workshop; May 2011; Vancouver p. 7-12.
- Wang H, Sun CT. Game Reward Systems: Gaming Experiences and Social Meanings. 2011 Presented at: 5th DiGRA Conference: Think Design Play; September 2011; Hilversum, The Netherlands p. 14-17.
- van Dam HA, van der Horst FG, Knoops L, Ryckman RM, Crebolder HF, van den Borne BH. Social support in diabetes: a systematic review of controlled intervention studies. Patient Educ Couns 2005 Oct;59(1):1-12. [CrossRef] [Medline]
- White M, Dorman SM. Receiving social support online: implications for health education. Health Educ Res 2001 Dec;16(6):693-707 [FREE Full text] [Medline]
- Mohr DC, Cuijpers P, Lehman K. Supportive accountability: a model for providing human support to enhance adherence to eHealth interventions. J Med Internet Res 2011;13(1):e30 [FREE Full text] [CrossRef] [Medline]
- van den Berg MH, Ronday HK, Peeters AJ, Voogt-van der Harst EM, Munneke M, Breedveld FC, et al. Engagement and satisfaction with an Internet-based physical activity intervention in patients with rheumatoid arthritis. Rheumatology (Oxford) 2007 Mar;46(3):545-552 [FREE Full text] [CrossRef] [Medline]
- Mohr DC, Duffecy J, Jin L, Ludman EJ, Lewis A, Begale M, et al. Multimodal e-mental health treatment for depression: a feasibility trial. J Med Internet Res 2010;12(5):e48 [FREE Full text] [CrossRef] [Medline]
- Vernmark K, Lenndin J, Bjärehed J, Carlsson M, Karlsson J, Oberg J, et al. Internet administered guided self-help versus individualized e-mail therapy: A randomized trial of two versions of CBT for major depression. Behav Res Ther 2010 May;48(5):368-376. [CrossRef] [Medline]
- Carrard I, Crépin C, Rouget P, Lam T, Golay A, Van der Linden M. Randomised controlled trial of a guided self-help treatment on the Internet for binge eating disorder. Behav Res Ther 2011 Aug;49(8):482-491. [CrossRef] [Medline]
- Spence J, Titov N, Dear BF, Johnston L, Solley K, Lorian C, et al. Randomized controlled trial of Internet-delivered cognitive behavioral therapy for posttraumatic stress disorder. Depress Anxiety 2011 Jul;28(7):541-550. [CrossRef] [Medline]
- Hedman E, Andersson G, Andersson E, Ljótsson B, Rück C, Asmundson GJ, et al. Internet-based cognitive-behavioural therapy for severe health anxiety: randomised controlled trial. Br J Psychiatry 2011 Mar;198(3):230-236 [FREE Full text] [CrossRef] [Medline]
- Gerrits RS, van der Zanden RAP, Visscher RFM, Conijn BP. Master your mood online: A preventive chat group intervention for adolescents. Advances in Mental Health 2007;6(3):152-111. [CrossRef]
- Perini S, Titov N, Andrews G. Clinician-assisted Internet-based treatment is effective for depression: randomized controlled trial. Aust N Z J Psychiatry 2009 Jun;43(6):571-578. [CrossRef] [Medline]
- Aydos L, Titov N, Andrews G. Shyness 5: the clinical effectiveness of Internet-based clinician-assisted treatment of social phobia. Australas Psychiatry 2009 Dec;17(6):488-492. [CrossRef] [Medline]
- Titov N, Andrews G, Choi I, Schwencke G, Mahoney A. Shyness 3: randomized controlled trial of guided versus unguided Internet-based CBT for social phobia. Aust N Z J Psychiatry 2008 Dec;42(12):1030-1040. [CrossRef] [Medline]
- Titov N, Andrews G, Schwencke G, Drobny J, Einstein D. Shyness 1: distance treatment of social phobia over the Internet. Aust N Z J Psychiatry 2008 Jul;42(7):585-594. [CrossRef] [Medline]
- Titov N, Andrews G, Schwencke G. Shyness 2: treating social phobia online: replication and extension. Aust N Z J Psychiatry 2008 Jul;42(7):595-605. [CrossRef] [Medline]
- van Wier MF, Ariëns GA, Dekkers JC, Hendriksen IJ, Smid T, van Mechelen W. Phone and e-mail counselling are effective for weight management in an overweight working population: a randomized controlled trial. BMC Public Health 2009;9:6 [FREE Full text] [CrossRef] [Medline]
- Ljótsson B, Falk L, Vesterlund AW, Hedman E, Lindfors P, Rück C, et al. Internet-delivered exposure and mindfulness based therapy for irritable bowel syndrome--a randomized controlled trial. Behav Res Ther 2010 Jun;48(6):531-539. [CrossRef] [Medline]
- Goessens BM, Visseren FL, de Nooijer J, van den Borne HW, Algra A, Wierdsma J, et al. A pilot-study to identify the feasibility of an Internet-based coaching programme for changing the vascular risk profile of high-risk patients. Patient Educ Couns 2008 Oct;73(1):67-72. [CrossRef] [Medline]
- McTigue KM, Conroy MB, Hess R, Bryce CL, Fiorillo AB, Fischer GS, et al. Using the internet to translate an evidence-based lifestyle intervention into practice. Telemed J E Health 2009 Nov;15(9):851-858. [CrossRef] [Medline]
- McCabe M, Price E. Internet-based psychological and oral medical treatment compared to psychological treatment alone for ED. J Sex Med 2008 Oct;5(10):2338-2346. [CrossRef] [Medline]
- Lorig KR, Ritter PL, Laurent DD, Plant K. Internet-based chronic disease self-management: a randomized trial. Med Care 2006 Nov;44(11):964-971. [CrossRef] [Medline]
- An LC, Klatt C, Perry CL, Lein EB, Hennrikus DJ, Pallonen UE, et al. The RealU online cessation intervention for college smokers: a randomized controlled trial. Prev Med 2008 Aug;47(2):194-199. [CrossRef] [Medline]
- Spence SH, Donovan CL, March S, Gamble A, Anderson RE, Prosser S, et al. A randomized controlled trial of online versus clinic-based CBT for adolescent anxiety. J Consult Clin Psychol 2011 Oct;79(5):629-642. [CrossRef] [Medline]
- Titov N, Andrews G, Schwencke G, Robinson E, Peters L, Spence J. Randomized controlled trial of Internet cognitive behavioural treatment for social phobia with and without motivational enhancement strategies. Aust N Z J Psychiatry 2010 Oct;44(10):938-945. [CrossRef] [Medline]
- March S, Spence SH, Donovan CL. The efficacy of an internet-based cognitive-behavioral therapy intervention for child anxiety disorders. J Pediatr Psychol 2009 Jun;34(5):474-487 [FREE Full text] [CrossRef] [Medline]
- Bennett GG, Herring SJ, Puleo E, Stein EK, Emmons KM, Gillman MW. Web-based weight loss in primary care: a randomized controlled trial. Obesity (Silver Spring) 2010 Feb;18(2):308-313 [FREE Full text] [CrossRef] [Medline]
- Buller DB, Woodall WG, Zimmerman DE, Slater MD, Heimendinger J, Waters E, et al. Randomized trial on the 5 a day, the Rio Grande Way Website, a web-based program to improve fruit and vegetable consumption in rural communities. J Health Commun 2008;13(3):230-249. [CrossRef] [Medline]
- Cussler EC, Teixeira PJ, Going SB, Houtkooper LB, Metcalfe LL, Blew RM, et al. Maintenance of weight loss in overweight middle-aged women through the Internet. Obesity (Silver Spring) 2008 May;16(5):1052-1060 [FREE Full text] [CrossRef] [Medline]
- Glasgow RE, Christiansen SM, Kurz D, King DK, Woolley T, Faber AJ, et al. Engagement in a diabetes self-management website: usage patterns and generalizability of program use. J Med Internet Res 2011;13(1):e9 [FREE Full text] [CrossRef] [Medline]
- Liebreich T, Plotnikoff RC, Courneya KS, Boulé N. Diabetes NetPLAY: A physical activity website and linked email counselling randomized intervention for individuals with type 2 diabetes. Int J Behav Nutr Phys Act 2009;6:18 [FREE Full text] [CrossRef] [Medline]
- Titov N, Andrews G, Choi I, Schwencke G, Johnston L. Randomized controlled trial of web-based treatment of social phobia without clinician guidance. Australian and New Zealand Journal of Psychiatry 2009;43(10):913-919. [CrossRef]
- Titov N, Andrews G, Schwencke G, Solley K, Johnston L, Robinson E. An RCT comparing effect of two types of support on severity of symptoms for people completing Internet-based cognitive behaviour therapy for social phobia. Australian and New Zealand Journal of Psychiatry 2009;43(10):920-926. [CrossRef]
- Woodall WG, Buller DB, Saba L, Zimmerman D, Waters E, Hines JM, et al. Effect of emailed messages on return use of a nutrition education website and subsequent changes in dietary behavior. J Med Internet Res 2007;9(3):e27 [FREE Full text] [CrossRef] [Medline]
- Thompson D, Baranowski T, Cullen K, Watson K, Canada A, Bhatt R, et al. Food, Fun and Fitness Internet program for girls: influencing log-on rate. Health Educ Res 2008 Apr;23(2):228-237 [FREE Full text] [CrossRef] [Medline]
- Stevens VJ, Funk KL, Brantley PJ, Erlinger TP, Myers VH, Champagne CM, et al. Design and implementation of an interactive website to support long-term maintenance of weight loss. J Med Internet Res 2008;10(1):e1 [FREE Full text] [CrossRef] [Medline]
- van Bastelaar K, Cuijpers P, Pouwer F, Riper H, Snoek FJ. Development and reach of a web-based cognitive behavioural therapy programme to reduce symptoms of depression and diabetes-specific distress. Patient Educ Couns 2011 Jul;84(1):49-55. [CrossRef] [Medline]
- Proudfoot J, Klein B, Barak A, Carlbring P, Cuijpers P, Lange A, et al. Establishing Guidelines for Executing and Reporting Internet Intervention Research. Cognitive Behaviour Therapy 2011;40(2):82-97. [CrossRef]
- van den Berg MH, Ronday HK, Peeters AJ, le Cessie S, van der Giesen FJ, Breedveld FC, et al. Using internet technology to deliver a home-based physical activity intervention for patients with rheumatoid arthritis: A randomized controlled trial. Arthritis Rheum 2006 Dec 15;55(6):935-945 [FREE Full text] [CrossRef] [Medline]
- Stinson JN, McGrath PJ, Hodnett ED, Feldman BM, Duffy CM, Huber AM, et al. An internet-based self-management program with telephone support for adolescents with arthritis: a pilot randomized controlled trial. J Rheumatol 2010 Sep;37(9):1944-1952. [CrossRef] [Medline]
- Lelieveld OT, Armbrust W, Geertzen JH, de Graaf I, van Leeuwen MA, Sauer PJ, et al. Promoting physical activity in children with juvenile idiopathic arthritis through an internet-based program: results of a pilot randomized controlled trial. Arthritis Care Res (Hoboken) 2010 May;62(5):697-703. [CrossRef] [Medline]
- Schulz PJ, Rubinelli S, Zufferey MC, Hartung U. Coping With Chronic Lower Back Pain: Designing and Testing the Online Tool ONESELF. J Comput Mediat Commun 2010;15(4):625-645. [CrossRef]
- Long AC, Palermo TM. Brief report: Web-based management of adolescent chronic pain: development and usability testing of an online family cognitive behavioral therapy program. J Pediatr Psychol 2009 Jun;34(5):511-516 [FREE Full text] [CrossRef] [Medline]
- Palermo TM, Wilson AC, Peters M, Lewandowski A, Somhegyi H. Randomized controlled trial of an Internet-delivered family cognitive-behavioral therapy intervention for children and adolescents with chronic pain. Pain 2009 Nov;146(1-2):205-213. [CrossRef] [Medline]
- Goessens BMB, Visseren FLJ, Geerts AC, Wierdsma J, van den Borne HW, Algra A, et al. Self-management of vascular patients activated by the Internet and nurses: Rationale and design. 2006 Presented at: Proceedings of the First International Conference on Persuasive Technology; 2006; Eindhoven, The Netherlands p. 18-19. [CrossRef]
- Mulvaney SA, Rothman RL, Wallston KA, Lybarger C, Dietrich MS. An internet-based program to improve self-management in adolescents with type 1 diabetes. Diabetes Care 2010 Mar;33(3):602-604. [CrossRef] [Medline]
- van Bastelaar KM, Pouwer F, Cuijpers P, Riper H, Snoek FJ. Web-based depression treatment for type 1 and type 2 diabetic patients: a randomized, controlled trial. Diabetes Care 2011 Feb;34(2):320-325. [CrossRef] [Medline]
- DiIorio C, Escoffery C, McCarty F, Yeager KA, Henry TR, Koganti A, et al. Evaluation of WebEase: an epilepsy self-management Web site. Health Educ Res 2009 Apr;24(2):185-197 [FREE Full text] [CrossRef] [Medline]
- McCabe MP, Price E. Attrition from an internet-based psychological intervention for erectile dysfunction: who is likely to drop out? J Sex Marital Ther 2009;35(5):391-401. [CrossRef] [Medline]
- Dew MA, Goycoolea JM, Harris RC, Lee A, Zomak R, Dunbar-Jacob J, et al. An internet-based intervention to improve psychosocial outcomes in heart transplant recipients and family caregivers: development and evaluation. J Heart Lung Transplant 2004 Jun;23(6):745-758. [Medline]
- Sorbi MJ, van der Vaart R. User acceptance of an Internet training aid for migraine self-management. J Telemed Telecare 2010;16(1):20-24. [CrossRef] [Medline]
- Lorig KR, Ritter PL, Dost A, Plant K, Laurent DD, McNeil I. The Expert Patients Programme online, a 1-year study of an Internet-based self-management programme for people with long-term conditions. Chronic Illn 2008 Dec;4(4):247-256. [CrossRef] [Medline]
- Andersson G, Strömgren T, Ström L, Lyttkens L. Randomized controlled trial of internet-based cognitive behavior therapy for distress associated with tinnitus. Psychosom Med 2002;64(5):810-816 [FREE Full text] [Medline]
- Buis LR, Poulton TA, Holleman RG, Sen A, Resnick PJ, Goodrich DE, et al. Evaluating Active U: an Internet-mediated physical activity program. BMC Public Health 2009;9:331 [FREE Full text] [CrossRef] [Medline]
- Thompson D, Baranowski J, Cullen K, Baranowski T. Development of a theory-based internet program promoting maintenance of diet and physical activity change to 8-year-old African American girls. Computers & Education 2007 Apr;48(3):446-459. [CrossRef]
- Baranowski T, Baranowski JC, Cullen KW, Thompson DI, Nicklas T, Zakeri IE, et al. The Fun, Food, and Fitness Project (FFFP): the Baylor GEMS pilot study. Ethn Dis 2003;13(1 Suppl 1):S30-S39. [Medline]
- Brouwer W, Oenema A, Raat H, Crutzen R, de Nooijer J, de Vries NK, et al. Characteristics of visitors and revisitors to an Internet-delivered computer-tailored lifestyle intervention implemented for use by the general public. Health Educ Res 2010 Aug;25(4):585-595 [FREE Full text] [CrossRef] [Medline]
- Brendryen H, Drozd F, Kraft P. A digital smoking cessation program delivered through internet and cell phone without nicotine replacement (happy ending): randomized controlled trial. J Med Internet Res 2008;10(5):e51 [FREE Full text] [CrossRef] [Medline]
- Brendryen H, Kraft P. Happy ending: a randomized controlled trial of a digital multi-media smoking cessation intervention. Addiction 2008 Mar;103(3):478-84; discussion 485. [CrossRef] [Medline]
- Lenert L, Muñoz RF, Stoddard J, Delucchi K, Bansod A, Skoczen S, et al. Design and pilot evaluation of an internet smoking cessation program. J Am Med Inform Assoc 2003;10(1):16-20 [FREE Full text] [Medline]
- Balmford J, Borland R, Benda P. Patterns of use of an automated interactive personalized coaching program for smoking cessation. J Med Internet Res 2008;10(5):e54 [FREE Full text] [CrossRef] [Medline]
- Graham AL, Cobb NK, Raymond L, Sill S, Young J. Effectiveness of an internet-based worksite smoking cessation intervention at 12 months. J Occup Environ Med 2007 Aug;49(8):821-828. [CrossRef] [Medline]
- Saul JE, Schillo BA, Evered S, Luxenberg MG, Kavanaugh A, Cobb N, et al. Impact of a statewide Internet-based tobacco cessation intervention. J Med Internet Res 2007;9(3):e28 [FREE Full text] [CrossRef] [Medline]
- Cobb NK, Graham AL, Bock BC, Papandonatos G, Abrams DB. Initial evaluation of a real-world Internet smoking cessation system. Nicotine Tob Res 2005 Apr;7(2):207-216. [CrossRef] [Medline]
- An LC, Perry CL, Lein EB, Klatt C, Farley DM, Bliss RL, et al. Strategies for increasing adherence to an online smoking cessation intervention for college students. Nicotine Tob Res 2006 Dec;8 Suppl 1:S7-12. [Medline]
- Hunter CM, Peterson AL, Alvarez LM, Poston WC, Brundige AR, Haddock CK, et al. Weight management using the internet: a randomized controlled trial. Am J Prev Med 2008 Feb;34(2):119-126. [CrossRef] [Medline]
- Meenan RT, Stevens VJ, Funk K, Bauck A, Jerome GJ, Lien LF, et al. Development and implementation cost analysis of telephone- and Internet-based interventions for the maintenance of weight loss. Int J Technol Assess Health Care 2009 Jul;25(3):400-410. [CrossRef] [Medline]
- Svetkey LP, Stevens VJ, Brantley PJ, Appel LJ, Hollis JF, Loria CM, et al. Comparison of strategies for sustaining weight loss: the weight loss maintenance randomized controlled trial. JAMA 2008 Mar 12;299(10):1139-1148. [CrossRef] [Medline]
- Funk KL, Stevens VJ, Appel LJ, Bauck A, Brantley PJ, Champagne CM, et al. Associations of internet website use with weight change in a long-term weight loss maintenance program. J Med Internet Res 2010;12(3):e29 [FREE Full text] [CrossRef] [Medline]
- Morgan PJ, Lubans DR, Collins CE, Warren JM, Callister R. 12-month outcomes and process evaluation of the SHED-IT RCT: an internet-based weight loss program targeting men. Obesity (Silver Spring) 2011 Jan;19(1):142-151 [FREE Full text] [CrossRef] [Medline]
- Titov N, Andrews G, Robinson E, Schwencke G, Johnston L, Solley K, et al. Clinician-assisted Internet-based treatment is effective for generalized anxiety disorder: randomized controlled trial. Australian and New Zealand Journal of Psychiatry 2009;43(6):905 [FREE Full text] [CrossRef]
- Titov N, Andrews G, Johnston L, Robinson E, Spence J. Transdiagnostic Internet treatment for anxiety disorders: A randomized controlled trial. Behav Res Ther 2010 Sep;48(9):890-899. [CrossRef] [Medline]
- Andersson G, Estling F, Jakobsson E, Cuijpers P, Carlbring P. Can the patient decide which modules to endorse? An open trial of tailored internet treatment of anxiety disorders. Cogn Behav Ther 2011 Mar;40(1):57-64. [CrossRef] [Medline]
- Linke S, Murray E, Butler C, Wallace P. Internet-based interactive health intervention for the promotion of sensible drinking: patterns of use and potential impact on members of the general public. J Med Internet Res 2007;9(2):e10 [FREE Full text] [CrossRef] [Medline]
- Linke S, Brown A, Wallace P. Down your drink: a web-based intervention for people with excessive alcohol consumption. Alcohol Alcohol 2004;39(1):29-32 [FREE Full text] [Medline]
- Postel MG, De Haan HA, De Jong CA. Evaluation of an e-therapy program for problem drinkers: a pilot study. Subst Use Misuse 2010 Oct;45(12):2059-2075. [CrossRef] [Medline]
- Postel MG, de Haan HA, ter Huurne ED, Becker ES, de Jong CA. Effectiveness of a web-based intervention for problem drinkers and reasons for dropout: randomized controlled trial. J Med Internet Res 2010;12(4):e68 [FREE Full text] [CrossRef] [Medline]
- Bingham CR, Barretto AI, Walton MA, Bryant CM, Shope JT, Raghunathan TE. Efficacy of a web-based, tailored, alcohol prevention/intervention program for college students: initial findings. J Am Coll Health 2010;58(4):349-356. [CrossRef] [Medline]
- Klein AA, Slaymaker VJ, Dugosh KL, McKay JR. Computerized continuing care support for alcohol and drug dependence: a preliminary analysis of usage and outcomes. J Subst Abuse Treat 2012 Jan;42(1):25-34. [CrossRef] [Medline]
- Schwinn TM, Schinke SP, Di Noia J. Preventing drug abuse among adolescent girls: outcome data from an internet-based intervention. Prev Sci 2010 Mar;11(1):24-32. [CrossRef] [Medline]
- Nicholas J, Proudfoot J, Parker G, Gillis I, Burckhardt R, Manicavasagar V, et al. The ins and outs of an online bipolar education program: a study of program attrition. J Med Internet Res 2010;12(5):e57 [FREE Full text] [CrossRef] [Medline]
- van Straten A, Cuijpers P, Smits N. Effectiveness of a web-based self-help intervention for symptoms of depression, anxiety, and stress: randomized controlled trial. J Med Internet Res 2008;10(1):e7 [FREE Full text] [CrossRef] [Medline]
- Warmerdam L, van Straten A, Twisk J, Riper H, Cuijpers P. Internet-based treatment for adults with depressive symptoms: randomized controlled trial. J Med Internet Res 2008;10(4):e44 [FREE Full text] [CrossRef] [Medline]
- de Graaf LE, Huibers MJ, Riper H, Gerhards SA, Arntz A. Use and acceptability of unsupported online computerized cognitive behavioral therapy for depression and associations with clinical outcome. J Affect Disord 2009 Aug;116(3):227-231. [CrossRef] [Medline]
- de Graaf LE, Gerhards SA, Arntz A, Riper H, Metsemakers JF, Evers SM, et al. Clinical effectiveness of online computerised cognitive-behavioural therapy without support for depression in primary care: randomised trial. Br J Psychiatry 2009 Jul;195(1):73-80 [FREE Full text] [CrossRef] [Medline]
- Meyer B, Berger T, Caspar F, Beevers CG, Andersson G, Weiss M. Effectiveness of a novel integrative online treatment for depression (Deprexis): randomized controlled trial. J Med Internet Res 2009;11(2):e15 [FREE Full text] [CrossRef] [Medline]
- Christensen H, Griffiths K, Groves C, Korten A. Free range users and one hit wonders: community users of an Internet-based cognitive behaviour therapy program. Aust N Z J Psychiatry 2006 Jan;40(1):59-62. [CrossRef] [Medline]
- Titov N, Dear BF, Schwencke G, Andrews G, Johnston L, Craske MG, et al. Transdiagnostic internet treatment for anxiety and depression: a randomised controlled trial. Behav Res Ther 2011 Aug;49(8):441-452. [CrossRef] [Medline]
- Carrard I, Fernandez-Aranda F, Lam T, Nevonen L, Liwowsky I, Volkart AC, et al. Evaluation of a guided internet self-treatment programme for bulimia nervosa in several European countries. Eur Eat Disord Rev 2010 Sep 21. [CrossRef] [Medline]
- Carlbring P, Nilsson-Ihrfelt E, Waara J, Kollenstam C, Buhrman M, Kaldo V, et al. Treatment of panic disorder: live therapy vs. self-help via the Internet. Behav Res Ther 2005 Oct;43(10):1321-1333. [CrossRef] [Medline]
- Carlbring P, Bohman S, Brunt S, Buhrman M, Westling BE, Ekselius L, et al. Remote treatment of panic disorder: a randomized trial of internet-based cognitive behavior therapy supplemented with telephone calls. Am J Psychiatry 2006 Dec;163(12):2119-2125. [CrossRef] [Medline]
- Farvolden P, Denisoff E, Selby P, Bagby RM, Rudy L. Usage and longitudinal effectiveness of a Web-based self-help cognitive behavioral therapy program for panic disorder. J Med Internet Res 2005;7(1):e7 [FREE Full text] [CrossRef] [Medline]
- Klein B, Richards JC, Austin DW. Efficacy of internet therapy for panic disorder. J Behav Ther Exp Psychiatry 2006 Sep;37(3):213-238. [CrossRef] [Medline]
- Richards JC, Klein B, Austin DW. Internet cognitive behavioural therapy for panic disorder: Does the inclusion of stress management information improve end-state functioning? Clinical Psychologist 2006;10(1):2-15. [CrossRef]
- Kiropoulos LA, Klein B, Austin DW, Gilson K, Pier C, Mitchell J, et al. Is internet-based CBT for panic disorder and agoraphobia as effective as face-to-face CBT? J Anxiety Disord 2008 Dec;22(8):1273-1284. [CrossRef] [Medline]
- Klein B, Austin D, Pier C, Kiropoulos L, Shandley K, Mitchell J, et al. Internet-based treatment for panic disorder: does frequency of therapist contact make a difference? Cogn Behav Ther 2009 Jun;38(2):100-113. [CrossRef] [Medline]
- Ruwaard J, Broeksteeg J, Schrieken B, Emmelkamp P, Lange A. Web-based therapist-assisted cognitive behavioral treatment of panic symptoms: a randomized controlled trial with a three-year follow-up. J Anxiety Disord 2010 May;24(4):387-396. [CrossRef] [Medline]
- Wims E, Titov N, Andrews G, Choi I. Clinician-assisted Internet-based treatment is effective for panic: A randomized controlled trial. Aust N Z J Psychiatry 2010 Jul;44(7):599-607. [CrossRef] [Medline]
- Klein B, Mitchell J, Abbott J, Shandley K, Austin D, Gilson K, et al. A therapist-assisted cognitive behavior therapy internet intervention for posttraumatic stress disorder: pre-, post- and 3-month follow-up results from an open trial. J Anxiety Disord 2010 Aug;24(6):635-644. [CrossRef] [Medline]
- Andersson G, Carlbring P, Holmström A, Sparthan E, Furmark T, Nilsson-Ihrfelt E, et al. Internet-based self-help with therapist feedback and in vivo group exposure for social phobia: a randomized controlled trial. J Consult Clin Psychol 2006 Aug;74(4):677-686. [CrossRef] [Medline]
- Tillfors M, Carlbring P, Furmark T, Lewenhaupt S, Spak M, Eriksson A, et al. Treating university students with social phobia and public speaking fears: Internet delivered self-help with or without live group exposure sessions. Depress Anxiety 2008;25(8):708-717. [CrossRef] [Medline]
- Furmark T, Carlbring P, Hedman E, Sonnenstein A, Clevberger P, Bohman B, et al. Guided and unguided self-help for social anxiety disorder: randomised controlled trial. Br J Psychiatry 2009 Nov;195(5):440-447 [FREE Full text] [CrossRef] [Medline]
- Tillfors M, Andersson G, Ekselius L, Furmark T, Lewenhaupt S, Karlsson A, et al. A randomized trial of internet-delivered treatment for social anxiety disorder in high school students. Cognitive Behaviour Therapy 2011;40(2):147-157. [CrossRef]
- Carlbring P, Gunnarsdóttir M, Hedensjö L, Andersson G, Ekselius L, Furmark T. Treatment of social phobia: randomised trial of internet-delivered cognitive-behavioural therapy with telephone support. Br J Psychiatry 2007 Feb;190:123-128 [FREE Full text] [CrossRef] [Medline]
- Hedman E, Andersson G, Ljótsson B, Andersson E, Rück C, Mörtberg E, et al. Internet-based cognitive behavior therapy vs. cognitive behavioral group therapy for social anxiety disorder: a randomized controlled non-inferiority trial. PLoS One 2011;6(3):e18001 [FREE Full text] [CrossRef] [Medline]
Edited by G Eysenbach; submitted 09.03.12; peer-reviewed by B Cugelman; comments to author 24.05.12; revised version received 27.05.12; accepted 27.06.12; published 14.11.12
©Saskia M Kelders, Robin N Kok, Hans C Ossebaard, Julia EWC Van Gemert-Pijnen. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 14.11.2012.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.