Published on 16.07.13 in Vol 15, No 7 (2013): July
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/2665, first published Apr 11, 2013.
Online Prevention Aimed at Lifestyle Behaviors: A Systematic Review of Reviews
Background: Interventions aimed at behavior change are increasingly being delivered over the Internet. Although research on intervention effectiveness has been widely conducted, their true public health impact as indicated by reach, effectiveness, and use is unclear.
Objective: The aim of this paper is to (1) review the current literature on online prevention aimed at lifestyle behaviors, and (2) identify research gaps regarding reach, effectiveness, and use.
Methods: A systematic search in PubMed revealed relevant literature published between 2005 and 2012 on Internet-delivered behavior change interventions aimed at dietary behaviors, physical activity, alcohol use, smoking, and condom use. Our search yielded 41 eligible reviews, which were analyzed in terms of reach, effectiveness, and use according to the RE-AIM framework.
Results: According to health priorities, interventions are largely targeted at weight-related behaviors, such as physical activity and dietary behavior. Evaluations are predominantly effect-focused and overall effects are small, variable, and not sustainable. Determinants of effectiveness are unclear; effectiveness cannot yet be unambiguously attributed to isolated elements. Actual reach of interventions is undiversified, mostly reaching participants who are female, highly educated, white, and living in high-income countries. One of the most substantial problems in online prevention is the low use of the interventions, a phenomenon seen across all behavior domains.
Conclusions: More research is needed on effective elements instead of effective interventions, with special attention to long-term effectiveness. The reach and use of interventions need more scientific input to increase the public health impact of Internet-delivered interventions.
J Med Internet Res 2013;15(7):e146
Unhealthy lifestyles have a major impact on morbidity and mortality . Health promotion is the process of enabling people to increase control over their health and its determinants, thereby improving their health [ ]. This also entails interventions targeting lifestyle behaviors. Health promotion is shifting its gaze toward new delivery modes (eg, the Internet) to effectively reach a larger part of the population with interventions targeting lifestyle behaviors. Access to the Internet is growing, especially in high-income areas, such as the United States (78.6%) and Europe (63.2%) [ ]. Therefore, Internet-delivered interventions have become a well-established instrument within the health promoter’s toolbox to potentially reach a large part of the population. Internet-delivered interventions can be operationalized as “typically behaviorally or cognitive behaviorally-based treatments that have been operationalized and transformed for delivery via the Internet”. Usually, they are highly structured; self or semi self-guided; based on effective face-to-face interventions; personalized to the user; interactive; enhanced by graphics, animations, audio, and possibly video; and tailored to provide follow-up and feedback” [ ].
The advantages of using the Internet as a delivery mode for health promotion are numerous. From a users’ point of view, the Internet is accessible 24/7 and interventions can be used anonymously and at any pace . Anonymity is favorable for interventions regarding behaviors that might involve shame (eg, condom use, alcohol moderation) [ ]. Internet-delivered interventions may reach populations who otherwise would not receive treatment [ ]. From a health promoters’ point of view, (1) the Internet offers improved opportunities for maintenance and updating of interventions [ ], (2) Internet-delivered interventions can mimic expensive face-to-face sessions in providing an individual as well as interactive approach in combination with an increased reach [ , ], and (3) Internet-delivered interventions are adjustable relatively easily to the needs of the user [ ]. The advantageous characteristics of Internet-delivered interventions are also related to the downsides. Anonymity and limited face-to-face contact, for instance, can lead to high attrition rates [ ]. This might affect the public health impact of these interventions.
Still, Internet-delivered interventions seem to possess potential with respect to health behavior change , but the actual public health impact remains unknown. Within the field of public health, intervention research is predominantly focused on intervention effectiveness [ ], or even on effective elements [ ]. This research focus oversimplifies reality in the quest to identify efficacious interventions. Effectiveness alone may reflect internal validity, but many interventions that prove to be effective in trials are much less effective when disseminated outside the context of a trial [ ]. It has become apparent that reach and use of interventions are at least as important as effectiveness, because the most effective intervention will not have a public health impact if its reach or actual use by the target group is low [ ].
The RE-AIM framework acknowledges that reach, effectiveness, adoption, implementation, and maintenance (hence, the acronym RE-AIM) are factors that all contribute to the public health impact of an intervention . Therefore, this study focuses on the public health impact of Internet-delivered interventions by taking all these factors into account [ ]. In the RE-AIM framework, reach is described as the percentage of individuals affected by a policy or program, but also the characteristics of those individuals [ ]. In this review, effectiveness is defined as changes in behavioral outcomes. Originally, the RE-AIM framework described adoption, implementation, and maintenance as factors at the organizational level. Internet-delivered interventions, however, can often be used standalone, which means that adoption, implementation, and maintenance also takes place (at least partly) at the individual level: each user decides whether to visit an intervention website for the first time (adoption), and whether to keep using it as intended (implementation), and for the long term (maintenance) [ , ]. These factors of the RE-AIM model are defined as use within the current study.
Within the field of public health, the use of the Internet as (the primary) delivery mode has expanded substantially and it is hard to imagine the public health field without the Internet. However, as mentioned previously, the public health impact as indicated by reach, effectiveness, and use remains unknown. This literature study comprises a systematic review of reviews addressing the following research questions:
- What is the reach, effectiveness, and use of Internet-delivered interventions aimed at lifestyle behaviors (ie, dietary behaviors, physical activity, alcohol use, smoking, and condom use)?
- What are the gaps in our current knowledge about the public health impact of Internet-delivered interventions aimed at lifestyle behaviors?
We identified relevant publications by conducting a PubMed search. The search query was designed in a way that both a search word regarding one of the behaviors of interest was in the title or abstract as well as a search word indicating the use of the Internet as (primary) delivery mode. Terms (including spelling variations and synonyms) that we searched for were eHealth OR Internet AND physical activity (exercise, sport, exertion, training, energy balance), smoking (tobacco, cigarette), alcohol (drinking, AOD, substance), nutrition (food, eat, weight, obesity, overweight, diet, adiposity), sexuality (safe sex, condom, HIV, AIDS, STI, STD), OR behavior (health, lifestyle, prevention, intervention). The exact search query can be found in.
The search was conducted in December 2012 and was limited to systematic reviews and meta-analyses published from 2005 to 2012 in the English language. Prior research was covered by a study conducted in 2005 by De Nooijer et al  in which no reviews were available.
Systematic reviews and meta-analyses were included if they (1) described at least 2 primarily Internet-delivered interventions aimed at behavior change regarding physical activity, smoking, alcohol use, dietary behaviors, or condom use, (2) reported on reach, effectiveness, or use of the included interventions, (3) were aimed at primary or secondary prevention for (part of) the general population, and (4) were available (full text) in English. Interventions aimed at health care workers or other intermediates were excluded. The article selection as well as data extraction was for all studies independently performed by 2 researchers (LK and RC) employing a conservative approach. If 1 of the 2 researchers was in doubt based on the title, the article was taken to the next round assessing the abstract. If 1 of the 2 researchers was in doubt based on the abstract, the article was taken to the next round assessing the full text. Disagreements were discussed until full consensus was reached.
The database search resulted in 4868 articles. Initial review was based on the title, after which 276 articles remained. Based on abstracts, 206 articles were excluded leaving 70 articles for a full-text analysis. Reasons for exclusion of articles based on full text (n=30) were that the publication was not a systematic review or meta-analysis (10/30, 33%), contained less than 2 eligible interventions (9/30, 30%), did not report measures on reach, effectiveness, or use of the interventions reviewed (5/30, 17%), were not primarily Internet-delivered (4/30, 13%), were aimed at treatment of a disease (1/30, 3%), or the full text was not available in the English language (1/30, 3%). Some articles were excluded for more than 1 reason. One article was retrieved in a manual search. Reference lists of the selected articles were checked for possible missed publications, but yielded no additional articles. Forty-one articles were found to be eligible for this review ().
An additional search was conducted in May 2013. This search extended the initial search by including the terms mhealth, smartphone, and mobile. In the initial search, these terms were not included because the focus of the study was on interventions that are primarily Internet-delivered. Therefore, reviews focusing on text messaging or apps were not included. Internet-delivered interventions are delivered by means of websites and it might indeed be that these are delivered by means of websites especially designed to be used/accessed on mobile phones. The additional search resulted in 174 hits (using the same limitations as the initial search). After title and abstract selection, there were 7 articles appropriate for a full-text analysis. Five of these concerned interventions that were not primarily Internet-delivered (eg, only text messaging). Two articles remained [, ], but these were already included in the initial search.
Data were extracted on the target group, number of eligible articles, and the size of corresponding study samples. Our main interest was what the studies reported on the reach, effectiveness, and use of the interventions reviewed. Reported limitations and recommendations were extracted also. Some of the selected reviews partly contained studies that did not correspond with the objectives of this study (eg, non–Internet-delivered interventions). Only data reflecting the eligible interventions have been reported and these data are included in.
A quality assessment was performed using the AMSTAR tool as well as the more detailed and sensitive R-AMSTAR tool. These tools are especially designed to assess the quality of systematic reviews and meta-analyses. With the AMSTAR tool, such studies are assessed on 11 quality criteria (yes/no/can’t answer/not applicable); total scores can range from 0 to 11 . The revised AMSTAR tool (R-AMSTAR) uses the same 11 criteria, but subdivides them into separate items, making the R-AMSTAR more sensitive. Total R-AMSTAR scores can range from 11 to 44 [ ]. Both these tools have been validated [ - ].
Characteristics of the Selected Studies
We included 41 papers [, , , , - ] after article selection ( ). Most studies were systematic reviews (27/41, 66%) and 16 reported meta-analyses (16/41, 39%) [ , , , , - , - , , ]. The studies were classified into 8 groups according to the behavior they targeted. The largest group targeted weight management, comprising 11 studies aimed at both dietary behaviors as well as physical activity [ , , , , , , , - , ]. All these were aimed at weight loss, and there were 5 reviews that also included interventions on weight maintenance [ , , , , ]. Six studies included 3 or more behaviors [ , , , - ]. The other groups included studies aimed at physical activity (6/41, 15%) [ , , , , , ], smoking (5/41, 12%) [ , , , , ], alcohol use (5/41, 12%) [ , , , , ], substance use including combined alcohol use and smoking (4/41, 10%) [ , , , ], and dietary behaviors (3/41, 7.3%) [ , , ]. An additional manual search revealed a study on condom use (1/41, 2.4%) [ ]. All studies were published between 2006 and 2012. Because we had broad inclusion criteria, study populations ranged from children aged 6 years [ ] to people aged 50 years and older [ ]. Most reviews were aimed at adults; 30 of 38 reviews (79%) reported age groups including adult populations. The mean of the AMSTAR score on overall quality of the included studies was 3.56 (SD 2.06). The mean of the R-AMSTAR score was 25.5 (SD 5.20). Item scores for all included studies are available in .
Most of the time, a very homogeneous sample of the population is reached. There is a strong female dominance, especially in weight loss/management interventions. Of the 18 reviews that reported gender distribution of the total sample, 16 reviews (89%) reported reaching more women than men [- , , , , , , , , , , , , ]. Participants were predominantly highly educated [ ], young, white [ , ], and living in high-income countries [ , ]. They were approached primarily through traditional offline recruitment efforts; however, some studies use online advertising or suggest to use more modern recruitment strategies [ ].
Overall, Internet-delivered interventions seem to have the potential to achieve behavior change. However, effect sizes were small, rarely moderate [, , , - , ]. Internet-delivered interventions compared to a no-treatment control condition had larger effect sizes than when compared with other interventions [ , , - ]. In some cases, often including face-to-face elements, the control was equally or more effective than the Internet-delivered intervention [ , , , , ]. It is not exactly clear what effective elements were and for whom these were effective [ , , ]. For some, indications of effectiveness were found. From this review, tailored feedback [ , , , , , ], use of theory [ , , ], interactivity [ , ], goal setting [ ], and combinations of online and in-person contact [ ] emerge as noteworthy promising constructs. It remains to be explored what elements work in what situation, and in what combination [ , , , - ]. Furthermore, it is also unclear when interventions become cost-effective [ , , , , - , , , ]. Long-term effects are measured in a limited number of interventions [ , , , , - ]; these effects are often unknown and the results that are available show very limited sustained effects (≥6 months, following the RE-AIM framework individual maintenance standards). Studies indicate that effect sizes decrease with intervention length and postintervention or that behavior is not maintained at all because effect sizes were quite small initially [ , , ]. Embedding an Internet-delivered intervention in existing structures (eg, schools, health care) might increase effectiveness [ , ]. Whether increased reach or use are the underlying driving forces behind this increased effectiveness is uncertain.
One of the largest problems in Internet-delivered interventions is low actual use. There is a wide variety of terminology used to describe use-related constructs (eg, adherence, exposure, and intervention attrition). Experimental research and theory development regarding intervention use is still in its infancy. Given the dose-response relationship between use and effectiveness, this is crucial to improve Internet-delivered interventions aimed at inducing behavior change. Factors suggested to stimulate the use of an intervention were sending reminders [, , ], incorporating professional support [ , , ], and embedding interventions in existing structures [ , ]. Process evaluations should explore people’s user experience in order to be able to adjust interventions accordingly [ ].
An illustrative description of the average Internet-delivered intervention states that a typical specimen is meant to be used once a week, is modular in setup, is updated once a week, lasts for 10 weeks, includes interaction with the system, a counselor, and peers on the Internet, includes some persuasive technology elements, and results in approximately 50% of the participants adhering to the intervention . Lastly, an issue regarding the use of an intervention is that this is mostly not objectively measured, and if so, heterogeneously, very poorly or not at all described [ , ].
The reviews on improving dietary behaviors primarily focused on younger populations (children, adolescents, young adults) [, ]. One study performed a cost-effectiveness analysis and concluded that eHealth devices are unlikely to be cost-effective [ ]. The studies showed small effects and the limited data on follow-up measures show that these effects were usually not maintained. The number of available reviews was small (3/41, 7.3%), and individual studies including follow-up measures were even scarcer (1/3, 33%). It was also recommended that the role of social support in Internet-delivered interventions shows promise and should be investigated more thoroughly [ ].
|Reference||Behavior||Study designa||AMSTAR scoreb||R-AMSTAR scorec|
|Harris et al (2011) ||Dietary behaviors||M||9||37|
|Hamel et al (2012) ||Dietary behaviors||SR||4||27|
|Lua et al (2012) ||Dietary behaviors||SR||1||18|
|Davies et al (2012) ||Physical activity||M||6||33|
|Lau et al (2011) ||Physical activity||SR||3||24|
|Laplante et al (2011) ||Physical activity||SR||4||26|
|Hamel et al (2011) ||Physical activity||SR||5||29|
|Van den Berg et al (2007) ||Physical activity||SR||2||24|
|Vandelanotte et al (2007) ||Physical activity||SR||1||23|
|Bewick et al (2008) ||Alcohol use||SR||3||25|
|Riper et al (2009) ||Alcohol use||M||5||31|
|Vernon (2010) ||Alcohol use||SR||1||18|
|White et al (2010) ||Alcohol use||SR||3||22|
|Riper et al (2011) ||Alcohol use||M||5||34|
|Myung et al (2009) ||Smoking||M||5||31|
|Shahab and McEwen (2009) ||Smoking||M||2||25|
|Civjlak et al (2010) ||Smoking||M||6||33|
|Hutton et al (2011) ||Smoking||SR||3||27|
|Chen et al (2012) ||Smoking||M||10||38|
|Noar et al (2009) ||Condom use||M||4||28|
|Weinstein et al (2006) ||Weight management||SR||1||18|
|Saperstein et al (2007) ||Weight management||SR||1||17|
|Maon et al (2012) ||Weight management||M||4||24|
|Fry et al (2009) ||Weight management||SR||2||20|
|An et al (2009) ||Weight management||SR||3||21|
|Neve et al (2010) ||Weight management||M||6||31|
|Manzoni et al (2011) ||Weight management||SR||2||18|
|Aalbers et al (2011) ||Weight management||SR||2||25|
|Kodama et al (2012) ||Weight management||M||5||32|
|Coons et al (2012) ||Weight management||SR||2||23|
|Norman et al (2007) ||Weight management||SR||2||22|
|Champion et al (2012) ||Substance use||SR||4||24|
|Lehto et al (2011) ||Substance use||SR||4||26|
|Rooke et al (2010) ||Substance use||M||2||23|
|Tait et al (2010) ||Substance use||M||3||23|
|Kelders et al (2012) ||Multiple behaviors||SR||3||24|
|Donkin et al (2011) ||Multiple behaviors||SR||1||21|
|Brouwer et al (2011) ||Multiple behaviors||SR||2||22|
|Crutzen et al (2011) ||Multiple behaviors||SR||4||27|
|Cugelman et al (2011) ||Multiple behaviors||M||5||27|
|Webb et al (2010) ||Multiple behaviors||M||3||23|
aM: meta-analysis; SR: systematic review.
bAMSTAR total score range 0-11.
cR-AMSTAR total score range 11-44.
Most reviews on physical activity conclude that when studies are effective, these show modest effect sizes with decreasing effectiveness during follow-up [, , , ]. A large meta-analysis showed that longer intervention duration, the use of social cognitive theory, the use of educational components, and regularly updating the content of the intervention had significant effects on physical activity levels [ ]. There was a reoccurring call for research on increasing intervention use [ , , ].
In alcohol abuse prevention, the risk profile of users is thought to be related to the effectiveness of the intervention . It is suggested that brief interventions may be more effective for high-risk participants than low-risk participants. Several studies focused on the use and content of feedback, in which findings on normative and personalized feedback seemed promising, but still inconclusive [ , ]. It was suggested to add a face-to-face component as an adjunct to Internet-delivered interventions to increase effectiveness, but effectiveness is not yet confirmed by research [ ]. Use is a problem [ , ]; 1 review showed that women, people married or living with a partner, and those without children were more likely to complete the program [ ]. Collecting user feedback to tailor interventions could increase this outcome [ ].
Almost all reviews aimed at smoking cessation recommended research that focuses on the use of the intervention, which is suggested to be improved by assessing barriers to participation . Effectiveness may be improved by involvement of users in the design of interventions [ , ]. Adolescents form a target group in need of extra attention [ , , ]. Biochemical measures to measure smoking behavior may be more accurate, but effectiveness appears to remain the same regardless of the method [ , , ]. Increasing interactivity and message tailoring seemed effective strategies [ , , ]. In smoking cessation, motivated participants, which were often more females than males, tended to show larger effects on behavior change [ ].
There was only 1 systematic review available on condom use, which discussed 3 Internet-delivered interventions . Individualized tailoring and the use of the stages of change model were related to effectiveness, but these were overall conclusions, not specified for Internet-delivered interventions.
In the weight management reviews, the majority of studies reported a strong overrepresentation of women (7/11, 64%), which makes a generalization of the effects to men questionable [, , ]. Evidence points toward the effectiveness of including a face-to-face component in Internet-delivered weight loss interventions. When personal contact is part of the intervention, results generally showed that effect sizes and use were higher [ , , , ]. When personal contact was not part of the intervention, reverse effects were seen [ ]. There was, however, no conclusive evidence. Interactive elements (eg, online peer support or forums) were shown to increase effectiveness; however, use was very limited [ ]. Some studies did not find a difference between Internet-delivered and traditional (face-to-face) interventions [ , , , , ]. It was suggested that a difference was hard to find because of the high attrition rates [ ] or that this finding might reflect an equal effectiveness, which could mean opportunities to increase interventions’ cost-effectiveness [ ]. Five reviews (5/11, 45%) included weight maintenance interventions in which the weight maintenance studies were always outnumbered by the weight loss studies [ , , , , ]. The limited findings indicate face-to-face interventions are more effective than Internet-delivered interventions.
Most studies in the reviews on substance use were aimed at adolescents or young adults, especially with regard to cigarette smoking behavior [, , ]. It is not clear what elements of the interventions were effective, but suggestions were made toward parental involvement, number of sessions, so-called booster sessions, normative education, resistance skills training, and reducing positive expectancies [ ]. Also within these behavior domains there was a demand for research that compares Internet-delivered with in-person interventions or combinations of the 2 [ , ]. One meta-analysis concluded that Internet-delivered interventions were significantly less effective than offline computer-delivered interventions [ ].
As opposed to the other review sections, effectiveness was not always the main outcome when multiple behaviors were considered. Reviews also focused on intervention use, which has been shown to be related to effectiveness. For instance, more adherent participants of weight loss programs lost more weight . Because effectiveness was the most commonly reported outcome, some reviews experienced difficulties collecting their data due to poor reporting on other constructs than effectiveness [ , , ]. Several reviews showed that nonadherence was a major problem of Internet-delivered interventions in particular, and a large barrier to effective interventions [ ]. Frequency of interaction (eg, email and/or phone contact with visitors, but also reminders), dialog support, updates [ ], targeted/tailored communication, monitoring of behavior change, professional and peer support, interactive and easy accessible content, conditional progress, incentives, and embedment in a social context all seemed to increase use [ ].
With regard to effectiveness, the reviews found that primary task support , extensive use of theory, inclusion of more behavior change techniques, elaborate interaction with participants, and training in stress management and general communication skills showed significant results on behavior change [ ]. Intervention duration was negatively correlated with impact [ ]. There was low use of interactive elements with peers and with professionals [ ].
The sizable growth of Internet-delivered interventions can be recognized when comparing the 41 reviews found in this study to the 9 individual studies found in a study with similar objectives conducted in 2005 . Almost half of the reviews on Internet-delivered interventions (20/41, 49%) were aimed at overweight-related behaviors (physical activity and dietary behaviors), reflecting a research focus congruent with health priorities [ ]. For smoking, the leading cause of preventable deaths in the United States [ ], 9 reviews were found. For condom use, data on Internet-delivered interventions were very limited; only 1 review was available [ ] focusing on computer-delivered interventions in general (ie, not necessarily Internet-delivered).
This review confirms the previously mentioned statement that within the public health field, there is a strong focus on effectiveness. It is reported most often and thoroughly, but effect sizes appear to be small, variable, and behavior change appears to be unsustainable at follow-up measures. Effect sizes may be small, but it should be noted that the Internet is a delivery mode with an unrivaled potential reach and this may still enhance public health impact . Estimates of actual reach for Internet interventions are lacking [ ]. The intended reach of Internet-delivered interventions is varied, aiming at a diverse population with respect to gender, socioeconomic status, and ethnic background, but the actual reach is undiversified; mostly the female, Caucasian, highly educated part of the population is reached, corresponding to previous findings [ ]. It is also desirable to reach high-risk participants; however, these are not necessarily attracted at the same rate as low-risk participants [ ]. Another limitation of the available studies is that most research is predominantly aimed at high-income populations [ , ], which makes generalization to low- and middle-income countries questionable. Although the potential reach of Internet-delivered interventions is virtually infinite, this review shows that, at least in terms of reach, there is still much to be gained.
Effect sizes are small, variable, and not sustainable. But what determines effectiveness? It is often not clear what the relevant active ingredients are for effectiveness [, ], and there is a clear call for elucidation on this subject from this review. The current data show indicative evidence for the effectiveness of tailored feedback, use of theory, interactivity, goal setting, and combinations of online and in-person contact. For several behavior change techniques, there seems to be some evidence, not all equally conclusive. Some well-substantiated evidence can be found for the effectiveness of message tailoring [ , ], but variations between interventions exist in the operationalization, and effectiveness is not guaranteed in all cases [ ]. A large meta-analysis found that extensive use of theory and the use of multiple behavior change techniques predict effectiveness in Internet-delivered interventions [ ]. Including face-to-face elements is recommended or found to increase the use or effectiveness in weight loss interventions [ , , ], alcohol use interventions [ ], and smoking cessation interventions [ ]. This could, however, have negative implications for reach. Including face-to-face support undermines the potentially high reach of Internet-delivered interventions, because of a high therapist burden. It is recommended to find a balance between face-to-face elements and self-guided Internet-delivered material [ , , , , ], also in the light of cost-effectiveness [ ]. Adjunctive designs have been applied to Internet-delivered interventions; however, it is not yet discerned what elements cause the effect of an intervention and whether these adjunctive elements contribute and to what extent [ ]. Some reviews indicated that interventions show improved effectiveness when they are embedded in an existing structure, such as schools or health care. This may not only improve effectiveness, but may improve reach and use as well. It is also suggested to use Internet-delivered interventions as a part of a stepped-care model [ ]. In this way, Internet-delivered interventions could serve as a first step in which individual needs are assessed with respect to the necessary amount of support, time, and expertise [ ].
Internet-delivered interventions are likely to be less costly than a face-to-face intervention and this is an oft-cited reason for delivering an intervention online . This reasoning probably holds for fully automated systems, but a fair point is made when considering that Internet-delivered interventions can still contain a substantial amount of human involvement [ ], which makes assumptions on cost-effectiveness less certain. Research on cost-effectiveness was recommended repeatedly in the reviews considered in the current study. A study answering this call concluded that cost-effectiveness is hard to determine, especially due to a lack of data [ ].
Data on the interventions’ use were poorly reported in most studies; in some cases, these data were completely lacking. These are missed opportunities, because Internet-delivered interventions in particular have the technological advantage to be able to provide more insight into intervention use . From the currently reported data, it is shown that there is particularly low use. The phenomenon defined as the discontinuation of eHealth application use, called the law of attrition, is considered to be “one of the fundamental characteristics and methodological challenges in the evaluation of eHealth applications” [ ]. In our own findings, the higher attrition rates in Internet-delivered compared to traditional interventions is most clearly illustrated by the review on condom use, in which the 2 trials with the poorest use were delivered online [ ]. Participants’ nonusage of an intervention can be explained from a reversed diffusion of innovations model [ , ]. The diffusion of innovations model is explained by a symmetric curve depicting the proportion of a population adopting an innovation and their motivations; the reversed model would reflect on the discontinuation of using an innovation (in this case, an intervention). Following this line of reasoning, factors influencing adoption may be used in Internet-delivered interventions to prevent low usage. There are a wide range of factors thought to stimulate intervention use. Christensen and Mackinnon [ ] point out the importance of user characteristics and preferences. A review of adherence in 3 Internet-delivered trials showed a positive association for higher self-efficacy, having less smoking friends, older age, being female, and a higher education with an increased use of the intervention [ ], but also interaction with a counselor, more frequent intended usage, more frequent updates, and more extensive employment of dialog support were found to be predictors of improved use [ ]. Engaging users is also thought to be improved by the transition from a more static, structured, developer-defined intervention content to less structured, more user-defined interactions [ , ]. Research on the use of Internet-delivered interventions is a relatively young and sometimes overshadowed endeavor; therefore, most findings are not yet confirmed by a substantive body of evidence. It is important that research is contributing to this debate [ ], because decreased intervention use has been shown to negatively affect health behavior change [ ] and the findings of the current study further underline the relevance of this problem.
Recommendations for Future Research
This review shows that a substantial amount of research has been done, but we found some reoccurring research needs, which will be discussed in terms of reach, effectiveness, and use.
The interventions’ reach is found to be undiversified and it is hard to reach high-risk groups. Participant recruitment can be done offline as well as online. Although reach seems to be much larger online, online recruitment can be a disappointing venture, even when targeting a young audience . Recruitment strategies should be aligned with users’ search strategies, which have to be studied first. Another strategy to increase reach is to create interest for the intervention. A lack of interest for Internet-delivered lifestyle interventions is identified by Lieberman and Massey [ ], who developed a motivational Internet application that was used to increase treatment interest in individuals with drinking problems not receiving treatment. Including a meta-intervention, which can be described as a small intervention prior to the actual intervention, there was increased participation of high-risk participants in an HIV-prevention intervention [ ]. It is also found that motivation for behavior change and curiosity regarding the content of the intervention appear to be important for a first visit to an Internet-delivered intervention [ ]. An extension of this line of research is warranted.
The largest gap in research on effectiveness seems to be that it is not known what intervention elements are effective and under what conditions. Future research should entail experimental studies focusing on testing isolated ingredients of Internet-delivered interventions. Effect evaluations on complete interventions are widely available and interventions proven to be effective could provide a starting point to disentangle effective ingredients. These may also be found outside the scope of Internet-based strategies, in adjunctive designs. Here, online and offline strategies are combined and it would be very interesting to investigate optimal combinations, possibly also in the context of cost-effectiveness. Interventions should include user profiles with information on a wide range of user characteristics. Such data are not only suitable for developing tailored advice, but could also shed light on effectiveness for subgroups of the population (eg, high-risk groups) to answer the question: what is effective for whom?
Discontinued or suboptimal use of the intervention is a widely recognized problem, also evident from the current review. There is a strong need for strategies that can be employed to increase use. User involvement in the creation of the intervention may be one of these strategies to keep users engaged. Some work has been done on this topic , but these findings need replication and further study. Moreover, there is a need for theory-driven experimental studies focusing on use of Internet-delivered interventions [ ].
For all areas of research, reviews recommend better data reporting because inadequate data reporting posits unnecessary limitations to research. This is especially the case when conducting a meta-analysis or review, or in the case of data on the reach or use of interventions. Full disclosure could be a solution, because it stimulates scrutiny and replication of research findings, and improves meta-analytical research [, ]. Moreover, the current technological developments should be employed more exhaustively. Therefore, cooperation with other disciplines is warranted to be able to develop modern, well-designed interventions. This intensified use of technology could not only be applied to data sharing, but also to increase effectiveness of interventions and to provide more insight into reach and use. Regarding the latter, there are some unused opportunities in terms of data collection [ ]. The Internet offers possibilities to collect a wide range of valuable data regarding intervention use (eg, log-ins, page views), which is readily available and should be disclosed.
Because of the substantial amount of data, we decided to perform a review of reviews. There are some weaknesses inherent to secondary analyses. It is inevitable that data and details are lost in order to obtain a more robust overview. The quality of our data is directly dependent on the reporting and execution quality of the reviewed studies. We performed a quality assessment on the included studies. The scores approached the lower end of the scales, but were comparable to those found in previous studies . Although the search was executed with the greatest consideration, it is possible that some publications or data may have been missed. This might be because only the primary search engine for this type of research was used to collect our data (ie, PubMed), although we also checked reference lists for completeness of our review, or because publications in English were solely considered in our review. Although we tried to be as elaborate as possible in the methodology, it should be acknowledged that this study may be affected by subjectivity bias because of the nature of the study design (ie, a systematic review of reviews) and quantitative data being available limitedly. Another possible bias lies in the fact that we did not correct for studies appearing in several of the included reviews. The result may be that some of the same evidence is covered more than once. Because we did not perform quantitative analyses, the resulting bias is estimated to be limited.
Despite these possible biases, we believe that the systematically collected data contributes to our understanding and to a general overview of what research has been done and what research still needs to be done.
This review provides an overview concerning research on online prevention aimed at lifestyle behaviors. The findings of this research show that reviews are effect-oriented, but interventions show small, unsustainable effects on behavior change. Research on reach and use of interventions is less advanced and needs more scientific input.
ZonMw (the Netherlands Organization for Health Research and Development) provided financial support to conduct this study.
Conflicts of Interest
Multimedia Appendix 1
Search query.PDF File (Adobe PDF File), 89KB
Multimedia Appendix 2
Description of included studies.PDF File (Adobe PDF File), 342KB
Multimedia Appendix 3
Quality assessment of included studies.XLSX File (Microsoft Excel File), 55KB
- Centers for Disease Control and Prevention. 2012. Chronic diseases and health promotion URL: http://www.cdc.gov/chronicdisease/overview/index.htm [accessed 2013-04-10] [WebCite Cache]
- World Health Organization. The Bangkok Charter for health promotion in a globalized world. Health Promot J Austr 2005 Dec;16(3):168-171. [Medline]
- Internet World Stats. 2012. World Internet usage and population statistics URL: http://www.internetworldstats.com/stats.htm [accessed 2013-04-10] [WebCite Cache]
- Ritterband LM, Thorndike F. Internet interventions or patient education web sites? J Med Internet Res 2006;8:e18. [CrossRef]
- Napolitano MA, Marcus BH. Targeting and tailoring physical activity information using print and information technologies. Exerc Sport Sci Rev 2002 Jul;30(3):122-128. [Medline]
- Moyer A, Finney JW. Brief interventions for alcohol problems: factors that facilitate implementation. Alcohol Res Health 2004;28(1):44-50. [Medline]
- Spek V, Cuijpers P, Nyklícek I, Riper H, Keyzer J, Pop V. Internet-based cognitive behaviour therapy for symptoms of depression and anxiety: a meta-analysis. Psychol Med 2007 Mar;37(3):319-328. [CrossRef] [Medline]
- Atkinson NL, Gold RS. The promise and challenge of eHealth interventions. Am J Health Behav 2002;26(6):494-503. [Medline]
- Neuhauser L, Kreps GL. Rethinking communication in the E-health era. J Health Psychol 2003 Jan;8(1):7-23. [CrossRef] [Medline]
- Cummins CO, Evers KE, Johnson JL, Paiva A, Prochaska JO, Prochaska JM. Assessing stage of change and informed decision making for Internet participation in health promotion and disease management. Manag Care Interface 2004 Aug;17(8):27-32. [Medline]
- Griffiths F, Lindenmeyer A, Powell J, Lowe P, Thorogood M. Why are health care interventions delivered over the internet? A systematic review of the published literature. J Med Internet Res 2006;8(2):e10 [FREE Full text] [CrossRef] [Medline]
- Eysenbach G. The law of attrition. J Med Internet Res 2005;7(1):e11 [FREE Full text] [CrossRef] [Medline]
- Cugelman B, Thelwall M, Dawes P. Online interventions for social marketing health behavior change campaigns: a meta-analysis of psychological architectures and adherence factors. J Med Internet Res 2011;13(1):e17 [FREE Full text] [CrossRef] [Medline]
- Sorensen G, Emmons K, Hunt MK, Johnston D. Implications of the results of community intervention trials. Annu Rev Public Health 1998;19:379-416. [CrossRef] [Medline]
- Webb TL, Joseph J, Yardley L, Michie S. Using the internet to promote health behavior change: a systematic review and meta-analysis of the impact of theoretical basis, use of behavior change techniques, and mode of delivery on efficacy. J Med Internet Res 2010;12(1):e4 [FREE Full text] [CrossRef] [Medline]
- Glasgow RE, Eakin EG, Toobert DJ. How generalizable are the results of diabetes self-management research? The impact of participation and attrition. Diabetes Educ 1996;22(6):573-574. [Medline]
- Bennett GG, Glasgow RE. The delivery of public health interventions via the Internet: actualizing their potential. Annu Rev Public Health 2009;30:273-292. [CrossRef] [Medline]
- Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999 Sep;89(9):1322-1327. [Medline]
- National Cancer Institute. 2012. Measuring the use of the RE-AIM model dimension items URL: http://www.re-aim.org/forms/re-aim_framework_constructs_checklist.pdf [accessed 2013-06-28] [WebCite Cache]
- Crutzen R, de Nooijer J, Brouwer W, Oenema A, Brug J, de Vries NK. Internet-delivered interventions aimed at adolescents: a Delphi study on dissemination and exposure. Health Educ Res 2008 Jun;23(3):427-439 [FREE Full text] [CrossRef] [Medline]
- Crutzen R, de Nooijer J, Brouwer W, Oenema A, Brug J, de Vries NK. A conceptual framework for understanding and improving adolescents' exposure to Internet-delivered interventions. Health Promot Int 2009 Sep;24(3):277-284 [FREE Full text] [CrossRef] [Medline]
- De Nooijer J, Oenema A, Kloek G, Brug H, De Vries H, De Vries N. Bevordering van gezond gedrag via internet, nu en in de toekomst [Promoting healthy behavior over the Internet, now and in the future]. Maastricht: Maastricht University; 2005.
- Lau PW, Lau EY, Wong del P, Ransdell L. A systematic review of information and communication technology-based interventions for promoting physical activity behavior change in children and adolescents. J Med Internet Res 2011;13(3):e48 [FREE Full text] [CrossRef] [Medline]
- Coons MJ, Demott A, Buscemi J, Duncan JM, Pellegrini CA, Steglitz J, et al. Technology interventions to curb obesity: A systematic review of the current literature. Curr Cardiovasc Risk Rep 2012 Apr;6(2):120-134. [CrossRef] [Medline]
- Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007;7:10 [FREE Full text] [CrossRef] [Medline]
- Kung J, Chiappelli F, Cajulis OO, Avezova R, Kossan G, Chew L, et al. From systematic reviews to clinical recommendations for evidence-based health care: Validation of revised assessment of multiple systematic reviews (R-AMSTAR) for grading of clinical relevance. Open Dent J 2010;4:84-91 [FREE Full text] [CrossRef] [Medline]
- Shea BJ, Bouter LM, Peterson J, Boers M, Andersson N, Ortiz Z, et al. External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS One 2007;2(12):e1350 [FREE Full text] [CrossRef] [Medline]
- Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, et al. AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol 2009 Oct;62(10):1013-1020. [CrossRef] [Medline]
- Aalbers T, Baars MA, Rikkert MG. Characteristics of effective Internet-mediated interventions to change lifestyle in people aged 50 and older: a systematic review. Ageing Res Rev 2011 Sep;10(4):487-497. [CrossRef] [Medline]
- Shahab L, McEwen A. Online support for smoking cessation: a systematic review of the literature. Addiction 2009 Nov;104(11):1792-1804. [CrossRef] [Medline]
- Weinstein PK. A review of weight loss programs delivered via the Internet. J Cardiovasc Nurs 2006;21(4):251-8; quiz 259. [Medline]
- Lua PL, Wan Putri Elena WD. The impact of nutrition education interventions on the dietary habits of college students in developed nations: a brief review. Malays J Med Sci 2012 Jan;19(1):4-14 [FREE Full text] [Medline]
- Riper H, Spek V, Boon B, Conijn B, Kramer J, Martin-Abello K, et al. Effectiveness of E-self-help interventions for curbing adult problem drinking: a meta-analysis. J Med Internet Res 2011;13(2):e42 [FREE Full text] [CrossRef] [Medline]
- Champion KE, Newton NC, Barrett EL, Teesson M. A systematic review of school-based alcohol and other drug prevention programs facilitated by computers or the Internet. Drug Alcohol Rev 2013 Mar;32(2):115-123. [CrossRef] [Medline]
- Chen YF, Madan J, Welton N, Yahaya I, Aveyard P, Bauld L, et al. Effectiveness and cost-effectiveness of computer and other electronic aids for smoking cessation: a systematic review and network meta-analysis. Health Technol Assess 2012;16(38):1-205 [FREE Full text] [CrossRef] [Medline]
- Davies CA, Spence JC, Vandelanotte C, Caperchione CM, Mummery WK. Meta-analysis of internet-delivered interventions to increase physical activity levels. Int J Behav Nutr Phys Act 2012;9:52 [FREE Full text] [CrossRef] [Medline]
- Kodama S, Saito K, Tanaka S, Horikawa C, Fujiwara K, Hirasawa R, et al. Effect of Web-based lifestyle modification on weight control: a meta-analysis. Int J Obes (Lond) 2012 May;36(5):675-685. [CrossRef] [Medline]
- Maon S, Edirippulige S, Ware R, Batch J. The use of web-based interventions to prevent excessive weight gain. J Telemed Telecare 2012 Jan;18(1):37-41. [CrossRef] [Medline]
- Myung SK, McDonnell DD, Kazinets G, Seo HG, Moskowitz JM. Effects of Web- and computer-based smoking cessation programs: meta-analysis of randomized controlled trials. Arch Intern Med 2009 May 25;169(10):929-937. [CrossRef] [Medline]
- Neve M, Morgan PJ, Jones PR, Collins CE. Effectiveness of web-based interventions in achieving weight loss and weight loss maintenance in overweight and obese adults: a systematic review with meta-analysis. Obes Rev 2010 Apr;11(4):306-321. [CrossRef] [Medline]
- Noar SM, Black HG, Pierce LB. Efficacy of computer technology-based HIV prevention interventions: a meta-analysis. AIDS 2009 Jan 2;23(1):107-115. [CrossRef] [Medline]
- Norman GJ, Zabinski MF, Adams MA, Rosenberg DE, Yaroch AL, Atienza AA. A review of eHealth interventions for physical activity and dietary behavior change. Am J Prev Med 2007 Oct;33(4):336-345 [FREE Full text] [CrossRef] [Medline]
- Riper H, van Straten A, Keuken M, Smit F, Schippers G, Cuijpers P. Curbing problem drinking with personalized-feedback interventions: a meta-analysis. Am J Prev Med 2009 Mar;36(3):247-255. [CrossRef] [Medline]
- Rooke S, Thorsteinsson E, Karpin A, Copeland J, Allsop D. Computer-delivered interventions for alcohol and tobacco use: a meta-analysis. Addiction 2010 Aug;105(8):1381-1390. [CrossRef] [Medline]
- Civljak M, Sheikh A, Stead LF, Car J. Internet-based interventions for smoking cessation. Cochrane Database Syst Rev 2010(9):CD007078. [CrossRef] [Medline]
- Hutton HE, Wilson LM, Apelberg BJ, Tang EA, Odelola O, Bass EB, et al. A systematic review of randomized controlled trials: Web-based interventions for smoking cessation among adolescents, college students, and adults. Nicotine Tob Res 2011 Apr;13(4):227-238. [CrossRef] [Medline]
- Manzoni GM, Pagnini F, Corti S, Molinari E, Castelnuovo G. Internet-based behavioral interventions for obesity: an updated systematic review. Clin Pract Epidemiol Ment Health 2011;7:19-28 [FREE Full text] [CrossRef] [Medline]
- Saperstein SL, Atkinson NL, Gold RS. The impact of Internet use for weight loss. Obes Rev 2007 Sep;8(5):459-465. [CrossRef] [Medline]
- Fry JP, Neff RA. Periodic prompts and reminders in health promotion and health behavior interventions: systematic review. J Med Internet Res 2009;11(2):e16 [FREE Full text] [CrossRef] [Medline]
- Hamel LM, Robbins LB. Computer- and web-based interventions to promote healthy eating among children and adolescents: a systematic review. J Adv Nurs 2013 Jan;69(1):16-30. [CrossRef] [Medline]
- Vandelanotte C, Spathonis KM, Eakin EG, Owen N. Website-delivered physical activity interventions a review of the literature. Am J Prev Med 2007 Jul;33(1):54-64. [CrossRef] [Medline]
- Tait RJ, Christensen H. Internet-based interventions for young people with problematic substance use: a systematic review. Med J Aust 2010 Jun 7;192(11 Suppl):S15-S21. [Medline]
- van den Berg MH, Schoones JW, Vliet Vlieland TP. Internet-based physical activity interventions: a systematic review of the literature. J Med Internet Res 2007;9(3):e26 [FREE Full text] [CrossRef] [Medline]
- Crutzen R, de Nooijer J, Brouwer W, Oenema A, Brug J, de Vries NK. Strategies to facilitate exposure to internet-delivered health behavior change interventions aimed at adolescents or young adults: a systematic review. Health Educ Behav 2011 Feb;38(1):49-62. [CrossRef] [Medline]
- White A, Kavanagh D, Stallman H, Klein B, Kay-Lambkin F, Proudfoot J, et al. Online alcohol interventions: a systematic review. J Med Internet Res 2010;12(5):e62 [FREE Full text] [CrossRef] [Medline]
- Kelders SM, Kok RN, Ossebaard HC, Van Gemert-Pijnen JE. Persuasive system design does matter: a systematic review of adherence to web-based interventions. J Med Internet Res 2012;14(6):e152 [FREE Full text] [CrossRef] [Medline]
- Brouwer W, Kroeze W, Crutzen R, de Nooijer J, de Vries NK, Brug J, et al. Which intervention characteristics are related to more exposure to internet-delivered healthy lifestyle promotion interventions? A systematic review. J Med Internet Res 2011;13(1):e2 [FREE Full text] [CrossRef] [Medline]
- Donkin L, Christensen H, Naismith SL, Neal B, Hickie IB, Glozier N. A systematic review of the impact of adherence on the effectiveness of e-therapies. J Med Internet Res 2011;13(3):e52 [FREE Full text] [CrossRef] [Medline]
- Harris J, Felix L, Miners A, Murray E, Michie S, Ferguson E, et al. Adaptive e-learning to improve dietary behaviour: a systematic review and cost-effectiveness analysis. Health Technol Assess 2011 Oct;15(37):1-160 [FREE Full text] [CrossRef] [Medline]
- Hamel LM, Robbins LB, Wilbur J. Computer- and web-based interventions to increase preadolescent and adolescent physical activity: a systematic review. J Adv Nurs 2011 Feb;67(2):251-268. [CrossRef] [Medline]
- Bewick BM, Trusler K, Barkham M, Hill AJ, Cahill J, Mulhern B. The effectiveness of web-based interventions designed to decrease alcohol consumption--a systematic review. Prev Med 2008 Jul;47(1):17-26. [CrossRef] [Medline]
- Vernon ML. A review of computer-based alcohol problem services designed for the general public. J Subst Abuse Treat 2010 Apr;38(3):203-211 [FREE Full text] [CrossRef] [Medline]
- Lehto T, Oinas-Kukkonen H. Persuasive features in web-based alcohol and smoking interventions: a systematic review of the literature. J Med Internet Res 2011;13(3):e46 [FREE Full text] [CrossRef] [Medline]
- An JY, Hayman LL, Park YS, Dusaj TK, Ayres CG. Web-based weight management programs for children and adolescents: a systematic review of randomized controlled trial studies. ANS Adv Nurs Sci 2009;32(3):222-240. [CrossRef] [Medline]
- Laplante C, Peng W. A systematic review of e-health interventions for physical activity: an analysis of study design, intervention characteristics, and outcomes. Telemed J E Health 2011 Sep;17(7):509-523. [CrossRef] [Medline]
- Cavallo DN, Tate DF, Ries AV, Brown JD, DeVellis RF, Ammerman AS. A social media-based physical activity intervention: a randomized controlled trial. Am J Prev Med 2012 Nov;43(5):527-532. [CrossRef] [Medline]
- Finkelstein EA, Khavjou OA, Thompson H, Trogdon JG, Pan L, Sherry B, et al. Obesity and severe obesity forecasts through 2030. Am J Prev Med 2012 Jun;42(6):563-570. [CrossRef] [Medline]
- Danaei G, Ding EL, Mozaffarian D, Taylor B, Rehm J, Murray CJ, et al. The preventable causes of death in the United States: comparative risk assessment of dietary, lifestyle, and metabolic risk factors. PLoS Med 2009 Apr 28;6(4):e1000058 [FREE Full text] [CrossRef] [Medline]
- Glasgow RE, McKay HG, Piette JD, Reynolds KD. The RE-AIM framework for evaluating interventions: what can it tell us about approaches to chronic illness management? Patient Educ Couns 2001 Aug;44(2):119-127. [Medline]
- Brouwer W, Oenema A, Raat H, Crutzen R, de Nooijer J, de Vries NK, et al. Characteristics of visitors and revisitors to an Internet-delivered computer-tailored lifestyle intervention implemented for use by the general public. Health Educ Res 2010 Aug;25(4):585-595 [FREE Full text] [CrossRef] [Medline]
- Glasgow RE, Nelson CC, Kearney KA, Reid R, Ritzwoller DP, Strecher VJ, et al. Reach, engagement, and retention in an Internet-based weight loss program in a multi-site randomized controlled trial. J Med Internet Res 2007;9(2):e11 [FREE Full text] [CrossRef] [Medline]
- Collins LM, Murphy SA, Nair VN, Strecher VJ. A strategy for optimizing and evaluating behavioral interventions. Ann Behav Med 2005 Aug;30(1):65-73. [CrossRef] [Medline]
- Norman GJ. Answering the "what works?" question in health behavior change. Am J Prev Med 2008 May;34(5):449-450 [FREE Full text] [CrossRef] [Medline]
- Noar SM, Benac CN, Harris MS. Does tailoring matter? Meta-analytic review of tailored print health behavior change interventions. Psychol Bull 2007 Jul;133(4):673-693. [CrossRef] [Medline]
- Kreuter MW, Farrell DW, Olevitch LR, Brennan LK. Tailoring Health Messages: Customizing Communication with Computer Technology. Mahwah, NJ: Lawrence Erlbaum; 1999.
- Kreuter MW, Oswald DL, Bull FC, Clark EM. Are tailored health education materials always more effective than non-tailored materials? Health Educ Res 2000 Jun;15(3):305-315 [FREE Full text] [Medline]
- Danaher BG, Seeley JR. Methodological issues in research on web-based behavioral interventions. Ann Behav Med 2009 Aug;38(1):28-39. [CrossRef] [Medline]
- Ritterband LM, Tate DF. The science of internet interventions. Introduction. Ann Behav Med 2009 Aug;38(1):1-3. [CrossRef] [Medline]
- Ritterband LM, Thorndike FP, Cox DJ, Kovatchev BP, Gonder-Frederick LA. A behavior change model for internet interventions. Ann Behav Med 2009 Aug;38(1):18-27 [FREE Full text] [CrossRef] [Medline]
- Tate DF, Finkelstein EA, Khavjou O, Gustafson A. Cost effectiveness of internet interventions: review and recommendations. Ann Behav Med 2009 Aug;38(1):40-45 [FREE Full text] [CrossRef] [Medline]
- Rogers E. Diffusion of Innovations. New York: Free Press; 2008.
- Christensen H, Mackinnon A. The law of attrition revisited. J Med Internet Res 2006;8(3):e20; author reply e21 [FREE Full text] [CrossRef] [Medline]
- Wangberg SC, Bergmo TS, Johnsen JA. Adherence in Internet-based interventions. Patient Prefer Adherence 2008;2:57-65 [FREE Full text] [Medline]
- Glasgow RE. Enhancing the scientific foundation of internet intervention research. Ann Behav Med 2009 Aug;38(1):46-47. [CrossRef] [Medline]
- Napolitano MA, Fotheringham M, Tate D, Sciamanna C, Leslie E, Owen N, et al. Evaluation of an internet-based physical activity intervention: a preliminary investigation. Ann Behav Med 2003;25(2):92-99. [Medline]
- Koo M, Skinner H. Challenges of internet recruitment: a case study with disappointing results. J Med Internet Res 2005;7(1):e6 [FREE Full text] [CrossRef] [Medline]
- Lieberman DZ, Massey SH. Pathways to change: the effect of a Web application on treatment interest. Am J Addict 2008;17(4):265-270. [CrossRef] [Medline]
- Albarracín D, Durantini MR, Earl A, Gunnoe JB, Leeper J. Beyond the most willing audiences: a meta-intervention to increase exposure to HIV-prevention programs by vulnerable populations. Health Psychol 2008 Sep;27(5):638-644. [CrossRef] [Medline]
- Brouwer W, Oenema A, Crutzen R, De Nooijer J, De Vries NK, Brug J. What makes people decide to visit and use an internet-delivered behavior-change intervention?: A qualitative study among adults. Health Education 2009;109:460-473.
- Crutzen R, Cyr D, de Vries NK. Bringing loyalty to e-Health: theory validation using three internet-delivered interventions. J Med Internet Res 2011;13(3):e73 [FREE Full text] [CrossRef] [Medline]
- Crutzen R, Peters GJ, Abraham C. What about trialists sharing other study materials? BMJ 2012;345:e8352. [Medline]
- Peters G-JY, Abraham C, Crutzen R. Eur Health Psychol. 2012. Full disclosure: doing behavioural science necessitates sharing URL: http://www.ehps.net/ehp/issues/2012/v14iss4_December2012/14_4_Peters,Abraham,Crutzen.pdf [accessed 2013-07-11] [WebCite Cache]
- Burda BU, Norris SL, Holmer HK, Ogden LA, Smith ME. Quality varies across clinical practice guidelines for mammography screening in women aged 40-49 years as assessed by AGREE and AMSTAR instruments. J Clin Epidemiol 2011 Sep;64(9):968-976. [CrossRef] [Medline]
Edited by G Eysenbach; submitted 11.04.13; peer-reviewed by F Drozd, A Preßler, S Desroches; comments to author 03.05.13; revised version received 28.06.13; accepted 04.07.13; published 16.07.13
©Leonie FM Kohl, Rik Crutzen, Nanne K de Vries. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 16.07.2013.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.