An Exploration of Factors Related to Dissemination of and Exposure to Internet-Delivered Behavior Change Interventions Aimed at Adults: A Delphi Study Approach

An Exploration of Factors Related to Dissemination of and Exposure to Internet-Delivered Behavior Change Interventions Aimed at Adults: A Delphi Study Approach

An Exploration of Factors Related to Dissemination of and Exposure to Internet-Delivered Behavior Change Interventions Aimed at Adults: A Delphi Study Approach

Original Paper

1Department of Public Health, Erasmus MC, University Medical Center, Rotterdam, The Netherlands

2Department of Health Education and Health Promotion, School for Public Health and Primary Care (Caphri), Maastricht University, Maastricht, The Netherlands

3EMGO Institute, VU University Medical Center, Amsterdam, The Netherlands

Corresponding Author:

Wendy Brouwer, MSc

Department of Public Health

Erasmus MC

PO Box 2040

3000 CA Rotterdam

The Netherlands

Phone: + 31 10 703 8468

Fax:+31 10 703 8475


Background: The Internet is an attractive medium for delivering individualized, computer-tailored behavior change interventions to large numbers of people. However, the actual numbers of people reached seem to fall behind the high expectations. Insight into factors that determine use of and exposure to these Internet interventions is important to be able to increase the reach and improve exposure.

Objective: The aim was to identify potentially important factors that determine whether adults visit an Internet-delivered behavior change intervention, extend their visit, and revisit the intervention.

Methods: A systematic, three-round Delphi study was conducted among national and international experts from Internet intervention research and practice, e-marketing/e-commerce, Web design, and technical website development. In the first round, 30 experts completed a structured, open-ended online questionnaire assessing factors that were, in their opinion, important for a first visit, an extended visit, a revisit and for effective promotion strategies. Based on the responses in this first questionnaire, a closed-ended online questionnaire was developed for use in the second round. A total of 233 experts were invited to complete this questionnaire. Median and interquartile deviation (IQD) scores were computed to calculate agreement and consensus on the importance of the factors. The factors for which no consensus was obtained (IQD > 1) were included in the third-round questionnaire. Factors with a median score of six or higher and with an IQD ≤ 1 were considered to be important.

Results: Of the 62 experts invited for the first round, 30 completed the questionnaire (48% response rate); 93/233 experts completed the second-round questionnaire (40% response rate), and 59/88 completed the third round (67% response rate). Being motivated to visit an Internet intervention and perceiving the intervention as personally relevant appeared to be important factors related to a first visit. The provision of tailored feedback, relevant and reliable information, and an easy navigation structure were related to an extended visit. Provision of regular new content and the possibility to monitor personal progress toward behavior change were identified as important factors to encourage a revisit. Primarily traditional promotion strategies, like word-of-mouth by family and friends, a publicity campaign with simultaneous use of various mass media, and recommendation by health professionals, were indicated as effective ways to encourage adults to visit an Internet intervention.

Conclusions: This systematic study identified important factors related to the dissemination of and exposure to Internet interventions aimed at adults. In order to improve optimal use of and exposure to Internet interventions, potential users may need to be motivated to visit such an intervention and the information provided needs to be personally relevant. Furthermore, several (technical) aspects of the intervention itself need to be taken into account when developing Internet interventions.

J Med Internet Res 2008;10(2):e10



The Internet has dramatically changed the possibilities for communication, including communication about health behavior and behavior change [1]. The Internet is a very attractive medium for the delivery of behavior change interventions since it provides the option of delivering sophisticated versions of individualized, computer-tailored interventions and holds the promise of reaching large numbers of people [2-5]. However, the actual reach of Internet-delivered behavior change interventions seems to lag behind this high expectation [6,7]. Evidence from efficacy trials indicates that actual use of and exposure to the assigned intervention content is low [8,9], and when implemented in real life, exposure rates may be even lower [10,11]. In addition, exposure to the intervention content is not always optimal. It has been demonstrated that it is difficult to sustain visitors’ loyalty to an intervention over an extended period of time [12,13], which may result in premature attrition from a session or in non-use of follow-up sessions. Furthermore, people tend to spend only a limited amount of time assessing the program [14], which makes optimal exposure to the intervention content unlikely. Loyalty to the program over an extended period of time may not be necessary for all Internet interventions or for all people using them since not all Internet interventions require extensive or repeated use of all the offered content [15,16]. However, for all Internet interventions at least some exposure to the intervention content is needed to initiate a process of behavior change. An increase in the number of people reached and improved exposure to Internet-delivered behavior change interventions are needed to be able to achieve optimal implementation of interventions after they have been evaluated to be efficacious [6,9].

The importance of focusing attention not only on intervention efficacy but also on dissemination, reach, and exposure in achieving public health impact is emphasized in the RE-AIM (Reach, Efficacy, Adoption, Implementation, and Maintenance) framework [17]. To be able to improve dissemination and exposure rates of Internet-delivered behavior change interventions, it is important to identify factors that enhance or inhibit these rates since such factors have to be targeted when attempting to improve dissemination and exposure [18]. The present study investigates factors related to dissemination of, use of, and exposure to Internet-delivered behavior change interventions among adults.

Access or use of the Internet is not likely to be a barrier to accessibility of Internet interventions these days since penetration rates of home Internet access and Internet use are high. Various factors have been related to Internet or Internet intervention use, for example, differences in motivation, skills, and availability of computer facilities [9,19]. It has been suggested that to increase the number of first time and extended visits, it is necessary to ensure reliability and credibility of the source or provider of the intervention [20,21]. The information structure has been found to be related to the use of information, with less structured websites tending to prematurely lose visitors [13,22,23]. Also, the amount of detail and elaboration of the information has been related to the length of time people process the intervention information [12]. Furthermore, it has been suggested that a static website that does not change over time may not attract revisits to interventions designed for multiple visits [4]. The use of email to encourage revisiting an intervention seemed to have some effect on revisits, but not on encouraging new users [7,24]. Even though some potentially important determining factors have been suggested in the literature, these factors have not been studied in a systematic way, which is the aim of the present study.

In this study we defined Internet-delivered behavior change interventions (or Internet interventions) to include those interventions that are aimed at the primary prevention of chronic disease by promoting healthful behaviors. Examples are interventions that promote healthful dietary, physical activity, and safe sex practices, discourage alcohol consumption, or encourage smoking cessation or sun protection behavior. Although these are very different topics, similar issues regarding exposure to and use of the content are likely to apply for all these interventions.

Dissemination and use of Internet interventions can be considered a process of diffusion and adoption of the intervention. Therefore, we used the Diffusion of Innovations Theory proposed by Rogers as the theoretical background for this study [18]. According to this model, characteristics of the user, the source (ie, the provider of the intervention), and the innovation (in this case the intervention) are important in the process of dissemination and adoption. Characteristics of the users include personal characteristics, such as gender and age, but also individual cognitions regarding use of Internet interventions, including attitudes, subjective norms, perceived behavioral control, and intention as derived from the Theory of Planned Behavior [25]. Furthermore, perceived possibilities and barriers to use of an intervention may play a role. Potentially important characteristics of the source are the perceived credibility and reliability. Characteristics of the intervention include the complexity (the degree to which the Internet intervention is perceived as difficult to understand and use), the trialability (the degree to which it is possible to experiment with the intervention before adopting it completely), and the relative advantage of the intervention (the degree to which the intervention is perceived to be superior to the idea that it replaces) [14,18]. In this study the term “dissemination” was used for the activities that the developers or providers have to undertake to bring the intervention to the attention of potential users. Dissemination was regarded as being distinct from exposure since the first is more related to activities of providers and the latter to the behavior of potential users. We conceptualized the process of visiting an Internet intervention and being optimally exposed to its educational content as consisting of three distinct phases that are potentially determined by different factors: (1) a first visit, in which a potential user has to decide to go to a website and see what it entails, (2) extending the visit, in which a user has to decide whether to continue his or her visit and be exposed to (part of) the content, and (3) revisiting the Internet intervention, in which the user has to decide to make a return visit to the intervention.

To assess the potential factors related to use of and exposure to Internet interventions, we conducted a three-round Delphi study. The specific aim of this study was to identify the (1) factors that are associated with dissemination of and exposure to (first visit, extended visit, and revisit) Internet interventions aimed at adults, and (2) extent to which experts agree on the importance of these factors.

A three-round Delphi study was conducted with international experts from health promotion research, e-marketing/ e-commerce, Web design, and technical website development. A Delphi study is a technique particularly suited for generating ideas about topics on which scientific knowledge is scarce. The technique allows for including experts from all over the world, guarantees anonymity of responses that may make the experts respond more freely, and is aimed at reaching agreement on the important issues [26-28]. The first round of the Delphi study was aimed at identifying potential factors of dissemination, first visit, extended visit, and revisit of an Internet intervention. The aim of the second and third round was to determine the importance and achieve agreement on the importance of the factors identified in the first round. The Delphi study was conducted over the Internet using online questionnaires. It was part of a larger study in which factors of dissemination and use of Internet interventions in adolescents were investigated. In the first round of the study, experts were asked to indicate factors that would be important for adults as well as for adolescents. In the second and third rounds, experts had to provide separate responses for adults and adolescents. The entire Delphi study was carried out within 3 months (October to December 2006). The results regarding adolescents are published elsewhere [29].

Participants and Procedure

A total of 62 prominent experts in Internet intervention research and practice, e-marketing/e-commerce, Web design, and technical website development from around the world were invited for the first round of the Delphi study. The ratio of experts from each field was set to 30:10:10:10. The highest number of experts was chosen to be from health promotion research and practice since we expected that these experts would have the broadest insight into the effectiveness of dissemination strategies and the factors related to a first visit, an extended visit, and a revisit. Criteria for choosing key experts in the first round were the following: (1) they were first authors of key scientific publications in the area of eHealth and eHealth promotion, and (2) they had written multiple scientific articles regarding this topic. People were also included if they were active members of editorial boards of leading journals in health promotion and the Internet and had published in these areas or journals. Representatives of e-marketing/e-commerce and ICT (information and communication technology) companies (eg, Web designers and developers) were selected on the basis of publications, our own network, and by asking the responders to provide names of other experts in their field.

This list of experts was extended to 233 persons (aim was 250) to be invited for participation in the second round of the study. The criterion for selection was being first author of a scientific paper or abstract on the topic of Internet interventions. Names of first authors were retrieved through a literature search in PubMed, PsycINFO, and Web of Science (between 2000 and 2006), and first authors of abstracts published in proceedings of relevant national and international conferences (eg, Society for the Internet in Medicine [MEDNET 2005 and 2006] and International Society for Behavior Nutrition and Physical Activity [ISBNPA 2004-2006]) were added to the list. Experts from the field of e-marketing/e-commerce and ICT were mainly found through our own network and by referral from experts in the first round. The experts who responded in the second round (n = 88) were invited to participate in the third round.

The experts were invited to participate in the study and each subsequent round by means of an email. In this email, the purpose and procedure of the Delphi study was explained and a link to the questionnaire was provided. Invitees were reminded once by email to complete the first-round questionnaire and twice to complete the second- and third-round questionnaires. The questionnaires were pre-tested by experts in the fields of health promotion research and e-marketing.


First Round

The first-round questionnaire was a structured questionnaire with an open-ended answer format. Participants were asked to list all the factors that, according to their expertise, (1) are essential for successful dissemination of Internet interventions, (2) determine whether a person will visit an intervention for the first time, (3) determine whether a person will stay long enough on a website to meaningfully engage in the educational content, and (4) determine whether a person will revisit a website. A sample question was “What are, according to your expertise, factors that determine whether a person will visit an Internet-delivered behavior change intervention for the first time?” The respondents were asked to suggest factors related to the user, the source, the Internet intervention itself, the physical and social environment, and any other important factors. The questionnaire started with a definition of all concepts used (eg, what we defined as factors, Internet-delivered interventions, behavioral topics addressed in these interventions, and dissemination).

Second Round

The second-round questionnaire had a closed-ended answer format and included all the unique factors that had been mentioned by the experts in the first round, except for those that were general health education principles not unique to Internet interventions (eg, the intervention is based on scientific knowledge, the information should be understandable) since these are basic principles for state of the art health communication interventions for which no rating of importance and consensus is needed. The questionnaire consisted of 82 statement items (see the Multimedia Appendix) presenting factors related to the (potential) visitor, the source, and the Internet intervention itself for a first visit, extended visit, revisit and for dissemination. The experts were asked to indicate how important they thought each of the factors were on a 7-point Likert scale (1 = not important, 7 = extremely important) for adults and adolescents separately. Apart from determinants of dissemination, the experts in the first round mentioned many factors that were, in fact, ways to promote Internet interventions. Therefore, we included a list with 23 strategies for promoting an Internet intervention. The experts were asked to choose the five strategies they thought were most successful for promoting an intervention among adults. This list of promotion strategies appeared in random order for each of the respondents.

Third Round

The third-round questionnaire contained the items (48 in total, see the Multimedia Appendix) of the second-round questionnaire for which no consensus was obtained (interquartile deviation [IQD] > 1). The answering scale for each item now included information on the median score and IQD for that item as determined in the second-round questionnaire. The experts were asked to re-rate their answers on the same 7-point Likert scale in the light of this new information.

Data Analysis

All the responses to the first-round questionnaire were listed, and similar responses were grouped together to reduce the number of factors. The remaining list of potentially important factors was included in the questionnaire for the second and third round, except for the factors that were general health education principles.

In the second round, following the standards for analyzing data from a Delphi study, the median scores were calculated to determine agreement on the importance of the statements. Also, the IQDs were calculated to determine consensus among the experts on the importance of the statements [26,30]. On a 7-point Likert scale, an IQD ≤ 1 can be considered as good consensus and means that more than 50% of all opinions fall within one point on the scale [28]. Items with a median ≥ 6 (very or extremely important) and an IQD ≤ 1 were considered as important factors. The dissemination strategies were analyzed by means of multiple response analysis.

In the third round, median scores and IQDs were calculated for the items included in the third-round questionnaire. SPSS 11.0 (SPSS Inc, Chicago, IL, USA) was used for all the statistical analyses.

Participants and Response Rates

In total, 30 of the 62 experts we approached completed the questionnaire in the first round (48% response rate; Table 1). Participants were primarily from health promotion institutes (64% response rate) and health promotion research (50% response rate); 93/233 respondents completed the second-round questionnaire (40% response rate), and 59/88 completed the third-round questionnaire (67% response rate). Three participants resigned from participation in the third round due to time constraints, and two could not be contacted again since they had not provided contact details in the previous questionnaire. Reasons for nonparticipation and dropout of the other experts are not known, although some reported lack of time or interest.

Table 1. Response rates in the Delphi study
DisciplineFirst RoundSecond RoundThird Round
No. InvitedNo. Responded%No. InvitedNo. Responded%No. InvitedNo. Responded%
Health promotion research3216501556542624166
Health promotion institutes1176420105010880
e-Marketing and communication9333246256467
Technical implementation1044034102910660


First Round

All factors unique for Internet interventions identified in the first round are listed in the Multimedia Appendix. This list is composed of factors that were mentioned by individual experts (eg, using modular approach, an enjoyable and rewarding experience in the first visit), as well as factors that were brought up by several of the experts (eg, tailored/individualized content, word-of-mouth by family and friends, the credibility of the source). More factors were mentioned for a first visit and an extended visit than for a revisit. The factors mentioned under dissemination were mainly ways to promote an intervention, such as word-of-mouth, commercials on TV and radio, and email.

Second Round

With respect to the first visit, 4 of 17 items pertaining to the potential visitor (sufficient Internet skills, experience with using the Internet, motivation to visit the intervention, perceived relevance of the intervention) and 2 of 9 items pertaining to the Internet intervention (instant use, easy navigation structure) had a median score ≥ 6 (Table 2). Consensus was reached for three of these items.

Regarding an extended visit, 5 of 9 items related to the visitor (eg, wants to improve behavior, experiences the use as rewarding, appreciates tailored feedback), 0 related to the source, and 12 of 23 items related to the Internet intervention (eg, displays personal progress, provides brief registration procedure, free of charge) had a median score ≥ 6 (Table 2). Consensus was reached for 10 of these items.

With respect to revisiting an intervention, 4 of 5 items regarding the visitor (receiving a reminder, committed to revisit, wants to improve behavior, positive experience with previous visit) and 5 of 10 items pertaining to the Internet intervention (new content, monitoring progress, experienced previous visit as easy, rewarding, and enjoyable) had a median score ≥ 6 (Table 2). Consensus was reached for all these items, indicating that the majority of experts agreed that these were important factors for revisiting.

None of the strategies for dissemination had a median score ≥ 6 (see the Multimedia Appendix).

Overall, consensus (IQD ≤ 1) was reached for 34 items in the second round. Most items that reached consensus were related to revisiting an intervention (10 of 15 items). The least consensus was achieved for dissemination of interventions (1 of 7 items).

Table 2. Results of the Delphi study per item (second and third round) with a median score ≥ 6 (full list of results including items with lower scores can be found in the Multimedia Appendix)
Questionnaire ItemSecond RoundThird Round
I. How important do you think each of the following factors are in determining whether a person will make a first visit to an Internet-delivered behavior change intervention?
A. Whether the potential visitor
- has sufficient skills to use the Internet8961.55961
- has experience with using the Internet8861*
- is motivated to visit a behavior change intervention provided through the Internet8861
- perceives the Internet intervention as relevant for himself/herself8461
B. Whether the Internet intervention
- can be used instantly without downloading special software by the potential visitor83625660
- has a navigation structure that appears to be easy to use at first sight83625660
II. How important do you think each of the following factors are in determining whether a person will stay on an Internet-delivered behavior change intervention long enough to actively engage in and process the educational content provided in the intervention?
A. Whether the visitor
- knows in advance how long it will take to go through the whole intervention80625661
- wants to improve his/her behavior in relation to the topic of the Internet intervention8061
- perceives the topic and content of the entire Internet intervention as being personally relevant79625660
- experiences the use of the Internet intervention as rewarding8061
- likes receiving (tailored) feedback on the answers he/she provided on questions80625661
C. Whether the Internet intervention
- displays personal progress through the program (eg, progress bar, page numbers)7861
- provides the opportunity for a visitor to stop at any moment and to proceed at a later time7961
- has an aim that is clear to the visitor7961
- provides information that appears reliable to the visitor7861
- provides information that is easy to understand for the visitor7961
- provides information that is perceived to be useful for the visitor to help him/her in changing behavior77625660
- has a tone of voice that is appealing to the visitor7861
- has an easy-to-follow navigation structure78625660
- provides tailored feedback7761
- provides tailored feedback which is perceived as relevant to the visitor7761
- provides behavior change information that seems achievable to the visitor77625660
- can be used free of charge77625560
III. How important do you think each of the following factors are in determining whether a person will revisit an Internet-delivered behavior change intervention?
A. Whether the visitor
- receives a reminder to revisit the Internet intervention7661
- is committed to revisiting the Internet intervention7661
- wants to improve his/her behavior in relation to the topic of the Internet intervention7661
- had a positive experience with the previous visit to the Internet intervention7661
B. Whether the Internet intervention
- provides new content on a regular basis7661
- provides the possibility for a visitor to monitor his/her progress in changing behavior7661
- has previously been experienced as easy to use by the visitor7661
- has previously been experienced as rewarding by the visitor7661
- has previously been experienced as enjoyable by the visitor7661

*Dashes indicate that consensus was obtained on the item in the second round and, for that reason, was excluded from the third-round questionnaire.

All items were scored 1-7 on a 7-point Likert scale.

The ways to disseminate Internet interventions that were indicated most often were word-of-mouth by family and friends (58.1%), a publicity campaign with the simultaneous use of various mass media (58.1%), and recommendation by health professionals (52.7%; Table 3).

Table 3. Strategies of Internet intervention dissemination (N = 74)
Dissemination StrategyNo.%
Word of mouth (eg, by friends and family)4358.1
Publicity campaign with simultaneous use of various mass media4358.1
Health professionals (eg, general practitioner, physical therapist)3952.7
TV and radio programs (eg, talk shows, consumer programs)3141.9
Commercials on TV and radio2837.8
Articles in magazines and newspapers2533.8
Links to the Internet intervention at other websites2027.0
Involvement of people who belong to the target group2027.0
Advertisements on websites visited by the target group1925.7
Face-to-face contact1824.3
Banners of the Internet intervention on other websites1418.9
Nonmedical professionals (eg, worksite health promoter)1418.9
Advertisements in magazines and newspapers1216.2
Advertisements on relevant products (eg, cigarette packs, milk cartons)1013.5
Free publicity (eg, postcards, brochures, bulletin board postings in libraries or hospitals)912.2
Use of virtual guides to direct people to the Internet intervention (eg, in chat boxes)810.8
Telephone calls79.5
Forums on the Internet45.4
Other ICT channels (eg, MSN Messenger, AOL Instant Messenger)34.1
Distribution of flyers at exhibitions and other public events22.7
Distribution of flyers door-to-door11.4
SMS (Short Message Service)00.0
Third Round

The median scores of the items included in the third-round questionnaire did not differ from the second round. Consensus was achieved for 45 of the 48 items (IQD ≤ 1; see Table 2). No consensus was achieved for positive expectations of behavior change interventions delivered through the Internet (relating to first visit), whether the user has to provide sensitive information, or the option of a trial before starting the intervention (related to extended visit). These three factors had a median score < 6.

Summary of Findings

This Delphi study is among the first systematic explorations of potentially important factors related to the dissemination of and exposure to Internet-delivered behavior change interventions. The study is unique in its focus on factors related to a first visit, an extended visit, and a revisit and by taking into account the characteristics of the potential users (in this case, adults), the source, and the intervention itself. In particular, factors related to the potential user, such as motivation and perceived personal relevance, were identified as important factors (median score > 6; IQD ≤ 1) related to a first visit. With regard to an extended visit (ie, staying on the intervention long enough to meaningfully process some of the content), many more factors related to the intervention itself were identified as important. The intervention needs to provide tailored feedback and relevant and reliable information and be clear and easy to use. The experience with the intervention in the previous visit, the inclination to change the behavior targeted in the intervention, the provision of new content, and being reminded to visit the intervention were regarded as important factors for a revisit. Apart from the factors that were rated as very important or extremely important, most of the other factors that came out of the first round reached consensus and were rated as somewhat important or important (median score 4-5). This means that these factors (listed in the Multimedia Appendix) also need to be taken into account when attempting to improve use and exposure to Internet interventions.

Interpretation of Findings

The existing knowledge on factors that enhance or inhibit optimal use of and exposure to an Internet intervention mainly relate to characteristics of the intervention itself. In this Delphi study we used the Diffusion of Innovations Theory [18] as a theoretical framework, and therefore, we also considered characteristics of the user and the source as potentially important factors associated with adoption. In contrast to previous studies, credibility and reliability of the source were not identified as very important factors for visiting an Internet intervention or extending a visit [20,21]. With respect to characteristics of the potential users, motivation to visit the intervention and perceived personal relevance of the intervention were identified as important factors. The finding that motivation is an important factor is intuitive since visiting an Internet intervention for the first time, extending the visit, and revisiting the intervention can be considered as specific behaviors that can be explained by the Theory of Planned Behavior [18]. According to this theory, motivation is the determinant most proximal to behavior. The present study did not, however, provide information about factors underlying the motivation to visit an Internet intervention, such as attitudes, subjective norms, or perceived behavioral control [18]. This is possibly due to the breadth of topics addressed in this study or that the study was performed among experts and not among the actual users of Internet interventions. Nevertheless, motivating people to visit an Internet intervention seems to be important.

The provision of personalized feedback seems to be a key element related to an extended visit to an Internet intervention. This finding underlines what has been previously suggested in the literature. Computer tailoring has been identified as a very promising health education technique and the Internet, as a suitable medium for delivery of computer-tailored interventions [31,32]. Furthermore, if the computer-tailored information is iterative and provides new information and information about the users’ progress, it might also encourage people to revisit the intervention [3,4,33-35].

Not only are motivation and personal feedback important, but the way in which the information is presented was also identified as an important factor for extending a visit and revisiting an Internet intervention. The navigation structure of the intervention must appear attractive and easy to use, as has been stressed before by Danaher et al [22]. Also, the intervention itself must look attractive at the very fist encounter (within 50 ms since an opinion about visual attractiveness is formed that quickly) [23]. Furthermore, the information obtained needs to be experienced as enjoyable and rewarding, but visitors must also find it usable and easy to understand [36].

An important factor to encourage people to revisit an Internet intervention that is designed for multiple visits is the provision of new content on a regular basis as there may be no need to return if the website does not change over time [4]. To make a revisit attractive, different aspects can be added to make the intervention less static, such as providing iterative tailored feedback or indicating what can be expected in a next visit. Another way to attract people to revisit the intervention is by reminding them, for example through email.

The communication channels most often indicated as potentially effective dissemination strategies were the more traditional channels such as word-of-mouth by family and friends [12], a publicity campaign with simultaneous use of various mass media, and recommendation by health professionals. Also, “old fashion” promotion strategies such as a publicity campaign, TV and radio commercials and programs, and articles in newspapers were seen as effective. The more novel channels, such as SMS, instant messaging, and banners on other websites, were hardly selected as important channels for dissemination.


There are some limitations to the study that need to be mentioned. We tried to incorporate experts from several disciplines as well as technical and marketing backgrounds. However, experts from technical and marketing backgrounds were underrepresented and responded less in the second and third round. Thus, the factors that were identified are more strongly based on the expert opinion of health educators and health promoters, and important factors from the technical and marketing field may have been missed. However, consensus was reached for most of the factors, which indicates that there were hardly any differences in the responses of experts from the various fields. Response rates in the various rounds ranged between 40% and 67%. Even though these response rates seem quite low, they are comparable to those found in other Delphi studies [26]. The low response rates may be due to the time investment that was required from the experts. They were asked to complete two or three questionnaires within 3 months. The low response rates may have resulted in the inclusion of a select group of experts, which may have introduced bias. We expect, however, that potential bias due to this selected sample is limited since the experts who participated provided a large variety of potentially important factors and saturation seemed to have been reached. Nevertheless, we cannot completely rule out the possibility that potentially important factors may have been missed. Most nonrespondents did not give a reason for not responding, but those who did mostly reported lack of time.

The Diffusion of Innovations Theory [18] and, within that, the Theory of Planned Behavior [25] that we used as a framework may not have been a complete fit for the present study and may have prevented us from looking at other potentially important factors. Another limitation may be that we tried to get information about various aspects of the process of visiting and revisiting an intervention. This breadth of topics may have been at the expense of the depth of information. The fact that mainly general factors were identified, such as “motivation” or “a rewarding experience,” and not factors that constitute motivation or a rewarding experience may be an indication of this. However, the aim of the present study was to gain a broad insight.

The results of the present study provide information about important factors for a first visit, extended visit, and a revisit that apply to most Internet-delivered behavior change interventions but that are not really intervention specific. Furthermore, not all factors identified in the present study may be equally applicable to all Internet interventions aimed at the primary prevention of chronic disease. That is because there is huge variety in the type of Internet intervention (low-intensity interventions without follow-up to very intensive interventions with up to 1 year follow-up), behavior targeted in the intervention, behavior change strategies applied, and so on. Therefore, for each intervention, the most applicable factors have to be chosen.


In this systematic exploration of potentially important factors determining whether adults visit an Internet-delivered behavior change intervention for the first time, extend a visit, and revisit the intervention, a number of factors were identified that can be taken into account when developing new Internet interventions. Further determinant research is needed to confirm the findings of this study and to identify important exposure-related factors from the perspective of the potential users.


This research was supported by a grant from ZonMw, Netherlands Organization for Health Research and Development (grant 4016.0017).

Conflicts of Interest

None declared.

Multimedia Appendix

Results of the Delphi study per item (second and third round)

PDF file (Adobe PDF), 108 KB

  1. Eng TR, Gustafson DH, Henderson J, Jimison H, Patrick K. Introduction to evaluation of interactive health communication applications. Science Panel on Interactive Communication and Health. Am J Prev Med 1999 Jan;16(1):10-15. [Medline] [CrossRef]
  2. Cassell MM, Jackson C, Cheuvront B. Health communication on the Internet: an effective channel for health behavior change? J Health Commun 1998;3(1):71-79. [Medline] [CrossRef]
  3. Marshall AL, Leslie ER, Bauman AE, Marcus BH, Owen N. Print versus website physical activity programs: a randomized trial. Am J Prev Med 2003 Aug;25(2):88-94. [Medline] [CrossRef]
  4. Napolitano MA, Fotheringham M, Tate D, Sciamanna C, Leslie E, Owen N, et al. Evaluation of an internet-based physical activity intervention: a preliminary investigation. Ann Behav Med 2003;25(2):92-99. [Medline] [CrossRef]
  5. Weinstein PK. A review of weight loss programs delivered via the Internet. J Cardiovasc Nurs 2006;21(4):251-8; quiz 259-60. [Medline]
  6. Glasgow RE. eHealth evaluation and dissemination research. Am J Prev Med 2007 May;32(5 Suppl):S119-S126. [Medline] [CrossRef]
  7. Leslie E, Marshall AL, Owen N, Bauman A. Engagement and retention of participants in a physical activity website. Prev Med 2005 Jan;40(1):54-59. [Medline] [CrossRef]
  8. Buller DB, Buller MK, Kane I. Web-based strategies to disseminate a sun safety curriculum to public elementary schools and state-licensed child-care facilities. Health Psychol 2005 Sep;24(5):470-476. [Medline] [CrossRef]
  9. De Nooijer J, Oenema A, Kloek G, Brug H, de Vries H, de Vries N. Bevordering van gezond gedrag via het Internet: nu en in de toekomst [Promotion of healthy behavior through the Internet: now and in the future]. Maastricht: Universiteit Maastricht; 2005.
  10. Evers KE, Prochaska JM, Prochaska JO, Driskell MM, Cummins CO, Velicer WF. Strengths and weaknesses of health behavior change programs on the Internet. J Health Psychol 2003;8(1). [CrossRef]
  11. Spittaels H, De Bourdeaudhuij I, Brug J, Vandelanotte C. Effectiveness of an online computer-tailored physical activity intervention in a real-life setting. Health Educ Res 2007 Jun;22(3):385-396. [Medline] [CrossRef]
  12. Anhøj J, Jensen AH. Using the internet for life style changes in diet and physical activity: a feasibility study. J Med Internet Res 2004 Sep 8;6(3):e28 [FREE Full text] [Medline] [CrossRef]
  13. Danaher BG, Boles SM, Akers L, Gordon JS, Severson HH. Defining participant exposure measures in Web-based health behavior change programs. J Med Internet Res 2006;8(3):e15 [FREE Full text] [Medline] [CrossRef]
  14. Eysenbach G. The law of attrition. J Med Internet Res 2005;7(1):e11 [FREE Full text] [Medline] [CrossRef]
  15. Christensen H, Mackinnon A. The law of attrition revisited. J Med Internet Res 2006;8(3):e20; author reply e21 [FREE Full text] [Medline] [CrossRef]
  16. Martinez M. High attrition rates in e-learning: challenges, predictors, and solutions.. The eLearning Developers Journal 2003.
  17. Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health 1999 Sep;89(9):1322-1327 [FREE Full text] [Medline]
  18. Rogers EM. Diffusion of Innovation. 4th edition. New York: The Free Press; 2003.
  19. ; Van Dijk JAGM. De digitale kloof wordt dieper [The Digital Gap Grows]. Zaandam: Kwak & van Daalen & Ronday; 2003.
  20. Brug J, Oenema A, Kroeze W, Raat H. The internet and nutrition education: challenges and opportunities. Eur J Clin Nutr 2005 Aug;59 Suppl 1:S130-7; discussion S138-9. [Medline] [CrossRef]
  21. Hong T. Contributing factors to the use of health-related websites. J Health Commun 2006 Mar;11(2):149-165. [Medline] [CrossRef]
  22. Danaher BG, McKay HG, Seeley JR. The information architecture of behavior change websites. J Med Internet Res 2005;7(2):e12 [FREE Full text] [Medline] [CrossRef]
  23. Lindgaard G, Fernandes G, Dudek C, Browñ J. Attention web designers: you have 50 milliseconds to make a good first impression!. Behav Inf Technol 2006;25(2). [CrossRef]
  24. Verheijden MW, Jans MP, Hildebrandt VH, Hopman-Rock M. Rates and determinants of repeated participation in a web-based behavior change program for healthy body weight and healthy lifestyle. J Med Internet Res 2007;9(1):e1 [FREE Full text] [Medline] [CrossRef]
  25. Ajzen I. Attitudes, Personality, and Behavior. Homewood, IL: Dorsey Press; 1988.
  26. De Vet E, Brug J, De Nooijer J, Dijkstra A, De Vries NK. Determinants of forward stage transitions: a Delphi study. Health Educ Res 2005 Apr;20(2):195-205 [FREE Full text] [Medline] [CrossRef]
  27. Green LW, Kreuter MW. Health Promotion Planning: An Educational and Ecological Approach. 3rd edition. Mountain View, CA: Mayfield; 1999.
  28. Linstone HA, Turoff M. The Delphi Method: Techniques and Applications. 2002.   URL: [accessed 2008 Mar 30] [WebCite Cache]
  29. Crutzen R, de Nooijer J, Brouwer W, Oenema A, Brug J, de Vries NK. Internet-delivered interventions aimed at adolescents: a Delphi study on dissemination and exposure. Health Educ Res 2008 Jan 21. [Medline] [CrossRef]
  30. Jones J, Hunter D. Consensus methods for medical and health services research. BMJ 1995 Aug 5;311(7001):376-380 [FREE Full text] [Medline]
  31. Kreuter M, Farrell D, Olevitch L, Brennan L. Tailoring Health Messages: Customizing Communication with Computer Technology. Mahwah, NJ: Erlbaum; 2000.
  32. Kroeze W, Werkman A, Brug J. A systematic review of randomized trials on the effectiveness of computer-tailored education on physical activity and dietary behaviors. Ann Behav Med 2006 Jun;31(3):205-223. [Medline] [CrossRef]
  33. Harvey-Berino J, Pintauro S, Buzzell P, Gold EC. Effect of internet support on the long-term maintenance of weight loss. Obes Res 2004 Feb;12(2):320-329. [Medline]
  34. Womble LG, Wadden TA, McGuckin BG, Sargent SL, Rothman RA, Krauthamer-Ewing ES. A randomized controlled trial of a commercial internet weight loss program. Obes Res 2004 Jun;12(6):1011-1018. [Medline]
  35. Wylie-Rosett J, Swencionis C, Ginsberg M, Cimino C, Wassertheil-Smoller S, Caban A, et al. Computerized weight loss intervention optimizes staff time: the clinical and cost results of a controlled clinical trial conducted in a managed care setting. J Am Diet Assoc 2001 Oct;101(10):1155-62; quiz 1163-4. [Medline] [CrossRef]
  36. Ferney SL, Marshall AL. Website physical activity interventions: preferences of potential users. Health Educ Res 2006 Aug;21(4):560-566 [FREE Full text] [Medline] [CrossRef]

ICT: information and communication technology
IQD: interquartile deviation
SMS: Short Message Service

Edited by G Eysenbach; submitted 13.07.07; peer-reviewed by L Ritterband, H Christensen, P Cuijpers; comments to author 28.11.07; revised version received 19.02.08; accepted 25.03.08; published 16.04.08


© Wendy Brouwer, Anke Oenema, Rik Crutzen, Jascha de Nooijer, Nanne K de Vries, Johannes Brug. Originally published in the Journal of Medical Internet Research (, 16.04.2008. Except where otherwise noted, articles published in the Journal of Medical Internet Research are distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided 1) the original work is properly cited, including full bibliographic details and the original article URL on, and 2) this statement is included.