JMIR J Med Internet Res Journal of Medical Internet Research 1438-8871 Gunther Eysenbach JMIR Publications Inc., Toronto, Canada v14i1e18 22357411 10.2196/jmir.1863 Original Paper Identifying Factors for Optimal Development of Health-Related Websites: A Delphi Study Among Experts and Potential Future Users Eysenbach Gunther Cugelman Brian Schneider Francine MSc 1
CAPHRI Department of Health Promotion Maastricht University P.O. Box 616 Maastricht, 6200 MD Netherlands 31 433882397 31 33671032 francine.schneider@maastrichtuniversity.nl
van Osch Liesbeth PhD 1 de Vries Hein PhD 1 1 CAPHRI Department of Health Promotion Maastricht University Maastricht Netherlands Jan-Feb 2012 14 02 2012 14 1 e18 07 06 2011 26 10 2011 25 11 2011 27 11 2011 ©Francine Schneider, Liesbeth van Osch, Hein de Vries. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 14.02.2012. 2012

This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.

 Background

The Internet has become a popular medium for offering tailored and targeted health promotion programs to the general public. However, suboptimal levels of program use in the target population limit the public health impact of these programs. Optimizing program development is considered as one of the main processes to increase usage rates.

 Objective

To distinguish factors potentially related to optimal development of health-related websites by involving both experts and potential users. By considering and incorporating the opinions of experts and potential users in the development process, involvement in the program is expected to increase, consequently resulting in increased appreciation, lower levels of attrition, and higher levels of sustained use.

 Methods

We conducted a systematic three-round Delphi study through the Internet. Both national and international experts (from the fields of health promotion, health psychology, e-communication, and technical Web design) and potential users were invited via email to participate. During this study an extensive list of factors potentially related to optimal development of health-related websites was identified, by focusing on factors related to layout, general and risk information provision, questionnaire use, additional services, and ease of use. Furthermore, we assessed the extent to which experts and potential users agreed on the importance of these factors. Differences as well as similarities among experts and potentials users were deduced.

 Results

In total, 20 of 62 contacted experts participated in the first round (32% response rate); 60 of 200 contacted experts (30% response rate) and 210 potential users (95% response rate) completed the second-round questionnaire, and 32 of 60 contacted experts completed the third round (53% response rate). Results revealed important factors consented upon by experts and potential users (eg, ease of use, clear structure, and detailed health information provision), as well as differences regarding important factors consented upon by experts (eg, visual aids, self-monitoring tool, and iterative health feedback) or by potential users only (eg, bread crumb navigation and prevention of receiving spam).

 Conclusions

This study is an important first step in determining the agreed-upon factors that should be taken into account when developing online health promotion programs. The public health impact of these programs will be improved by optimizing the development process in line with these factors.

 Health promotion Internet interventions health behavior change optimal development Delphi study experts users Introduction

Worldwide, more and more people are accessing the Internet in search of health-related information [1]. It is estimated that globally a minimum of nearly seven million health-related Internet searches are conducted daily [2]. Since Internet penetration rates are still expanding, with currently almost two billion people having access to the Internet, the number of health-related searches is also expected to increase [3]. Therefore, the Internet is a promising channel for offering a broad range of health-related information, such as background information on health, treatment information, medication information, and health behavior information [1].

Due to this high level of accessibility and its potential to reach large numbers of people [3], the Internet has also become a popular medium in the field of health promotion for offering tailored and targeted health promotion programs [4,5]. As a consequence, in recent years positive effects of online interventions applying computer-tailored techniques have been reported addressing different health behaviors [6], such as physical activity [7,8], nutrition [9,10], smoking cessation [11-14], and alcohol consumption [15,16]. Although these tailored interventions are very promising and have proven to be effective, actual reach is failing to live up to the high expectations [17-22]. Since the public health impact of interventions is determined not only by their efficacy but also by their levels of exposure in the target group [23,24], it is imperative to put effort into optimizing the level of exposure to Internet-delivered lifestyle interventions.

Successful exposure is partly defined by the level of first-time access of the intervention, also referred to as first use or adoption ([19,24,25]. Besides first use, prolonged use of the intervention is essential. That is, engaging users in the intervention for a substantial amount of time fosters their knowledge of its content and involvement in its effective components, which consequently increases the chances of health behavior change [17,26]. Since health behavior change is a complex and continuous process, achieving sustained behavior change depends on both the intensity of the intervention and the number of times the intervention is visited [27,28]. Due to this high dose–response relationship, ensuring adherence or sustained use of the program is essential to further maximize its effect on subsequent health behavior change [29,30]. Both prolonged and sustained use of the intervention can be influenced not only by user characteristics (eg, demographic characteristics and motivation to use the intervention [17,31,32]) but also by specific strategies to increase adherence (eg, sending periodic reminders [33,34]) and by intervention characteristics (eg, appearance and content of the intervention [33,35]).

As online behavior change interventions are delivered to the public by using a website or Web-based program, development of the website or program refers to composition of the actual intervention, and requires careful composition of the complete website or program it is embedded in. The website as a mode of delivery is described in the Internet intervention model as consisting of 8 main areas [35]: appearance (eg, the organization of information), behavioral prescriptions (eg, instructions on how to achieve behavior change), burdens (eg, poor navigation applications), content (eg, treatment information), delivery (eg, use of animations, audio, or graphics), message (eg, credibility and likability of the source), participation (eg, degree of interaction or the use of rewards), and assessment (eg, measuring needs of users or adjusting content to personal wishes). Adjusting these characteristics enables tailoring of the website to special needs of the population under consideration. All individual characteristics in these areas should therefore be kept in mind while developing the program.

The primary characteristics of an intervention are determined during its development process. According to diffusion theory and social marketing principles [25,36], program development is one of the main processes to influence adoption rates of a new product (eg, a new website or online intervention). Besides including experts in the development process, it should also be done in accordance with the needs and wishes of the target group [36]. By considering and incorporating the opinions of potential future users, involvement is expected to increase, consequently resulting in increased appreciation of the intervention, lower levels of attrition, and higher levels of prolonged and sustained use [37]. A new intervention should, therefore, be developed in close collaboration with the target group.

Although many studies have investigated the effectiveness and appreciation of numerous components of websites that deliver health behavior interventions, results have been inconclusive. Some studies recommend using a multimedia approach [38] or the use of interactive tools, such as surveys, quizzes, and games, whereas other studies seem to contradict these findings [5]. Systematically studying different elements of websites is a very extensive and time-consuming process. Even though some elements of websites should be incorporated at all times, such as accurate and comprehensible information [39], a lot of effort should still be put into making other elements operational [35].

To date, studies on development of health-related websites have not included perspectives of both experts and users. Hence, this study was a first attempt to identify which elements on health-related websites are perceived as necessary and preferable by these two groups. The current study thus included both experts in the fields of health promotion, health psychology, e-communication, and technical Web design and potential future users. By including elements that are perceived as important by both groups while developing the website, developers may optimize exposure rates of the program [36]. Furthermore, involving different groups of experts will lead to more diverse information regarding health communication and behavior change as well as technical information regarding website development. To identify the potential factors that are related to optimal development of health-related websites, we conducted a three-round Delphi study. The specific aim of the study was to identify factors that are associated with optimal development of health-related websites. Besides identifying these factors, we investigated the degree to which experts and potential future users agreed on the importance of the factors.

 Methods

We conducted a three-round Delphi study through the Internet. Due to its systematic nature, a Delphi study is considered to be an appropriate method to derive consensus on health-related issues for which scientific evidence is incomplete or scarce by involving a representative panel of experts [40]. Also, because after each round feedback on group results is provided, the iterative approach allows participants to adjust their opinions when needed. Finally, the structure of the study guarantees anonymity of the participants, thereby preventing conformity biases [41,42].

The current Delphi study consisted of two substudies. The first substudy (study 1a), including only experts, consisted of three rounds. The first round aimed at providing a list of potential factors related to optimal development of health-related websites by means of an open-ended questionnaire. Next, for the second round, experts were invited to rate the importance of all factors identified in the first round by using a structured questionnaire. Finally, a third round enabled experts to reevaluate their opinions by providing controlled feedback regarding group mean scores, thereby producing consensus. The second substudy (study 1b) included only potential future users and consisted of one round. This round resembled the second round of the first substudy and allowed users to rate the importance of those factors that were identified by experts in the first round by using the same structured questionnaire. We compared second-round results from both studies in order to identify potential differences between experts and potential users regarding the importance they placed on factors related to optimal design.

Study 1a: Experts First Round Procedure and Participants

For the first round, we selected experts from the fields of health promotion, health psychology, e-communication, and technical Web design to obtain a variety of insights from researchers with both theory-based and more practice-based backgrounds. Invited experts with a theory-based background were all first or second authors on scientific papers in the field of eHealth and eHealth promotion published between 2000 and 2009. We used database searches in PsychINFO and Medline to identify experts and to examine reference lists from related papers, book chapters, review studies, and conference abstracts. We selected experts with a practice-based background on the basis of their publications, but also by approaching our own network and by asking responding experts to provide names of important experts in the field.

This resulted in a list of 62 experts who were invited by email to participate in all three rounds of the Delphi study. The email contained detailed information on the goal and study procedure, as well as a link referring them directly to the first-round questionnaire. Nonresponders received a reminder email after the 3-week response period expired. A total of 20 experts (32% response rate) responded to the invitation.

 Questionnaire

The first-round questionnaire consisted of 7 open-ended questions. The questions pertained to different subjects related to the development of health-related websites: (1) optimal layout, (2) type of general information provided, (3) type of health risk information provided, (4) ease of use, (5) use of visual aids, and (6) additional information provided. Health-related websites often provide questionnaires to allow visitors the opportunity to assess their own current health status. To provide an accurate update of their health status these questionnaires often tend to be extensive and therefore sensitive to early dropout. Therefore, we also included one open-ended question to gain more insight into factors that contribute to completion of the questionnaires often provided on health-related websites (7). Health-related websites were defined as websites aiming at assisting people to adopt a healthier lifestyle, by offering them important and diverse information regarding health and health-related behaviors (eg, physical activity, smoking, alcohol consumption, and nutrition).

 Data Analysis

Responses of experts were analyzed, resulting in an extensive list of potential factors related to optimal development of health-related websites. Two researchers independently listed all unique factors and combined similar responses into 1 factor. For those factors on which no agreement was obtained, a third researcher was approached to give a decisive answer.

 Second Round Procedure and Participants

Experts participating in the first round were also invited to participate in the second round. We selected an additional 180 experts by means of the same strategies used for the first round.

A total of 200 experts received an email inviting them to participate in the second and third rounds. The invitation contained a link that directed experts to the second-round questionnaire. Nonresponders received two reminder emails: after 3 and after 5 weeks. A total of 60 experts (30% response rate) responded to the invitation (Table 1).

First-, second-, and third-round rates of responses of experts

Field First round (n = 62) Second round (n = 200) Third round (n = 60)
Invitations Response (%) Invitations Response (%) Invitations Response (%)
Health promotion 16 7 (44%) 60 18 (30%) 18 10 (56%)
Health psychology 16 5 (31%) 60 14 (23%) 14 8 (57%)
E-communication 15 5 (33%) 40 12 (30%) 12 5 (42%)
Technical Web design 15 2 (13%) 40 16 (40%) 16 9 (56%)
Total 62 20 (32%) 200 60 (30%) 60 32 (53%)
Questionnaire

The second-round questionnaire was composed based on the factors identified by experts in the first round. This resulted in a questionnaire consisting of 85 structured questions. Experts were asked to rate these factors on a 7-point Likert scale ranging from 1 (not at all important) to 7 (extremely important).

 Data Analysis

The closed-ended questions were analyzed following the standards for analyzing data for a Delphi study, by calculating median scores, also referred to as the 50th percentile score, to determine the importance of the various factors. Furthermore, the interquartile range (IQR) was calculated to assess the degree of agreement between the experts on the importance of the factors [31,41,43]. The IQR represents the distance between the 25th and the 75th percentiles, with a small value indicating a higher degree of agreement. An IQR smaller than 1 is considered to indicate good consensus on a 7-point scale as used in the present study and means that more than 50% of all cases fall within 1 point of one another [44]. To deduce those factors that were considered to be either very or extremely important by the majority of experts (IQR < 1), the cut-off point for importance was a median score of ≥6.

 Third Round Procedures and Participants

All experts participating in the second round (n = 60) were invited to participate in the third and final Delphi round, using the same procedure as in the previous round. A total of 32 experts (53% response rate of second-round participants) responded to the invitation (Table 1).

 Questionnaire

The third-round questionnaire was an adapted version of the second-round questionnaire, containing only those factors (n = 27) on which no consensus was obtained during the second round (IQR > 1). In line with the Delphi method, additional feedback on second-round group results (median and IQR) was provided, enabling experts to rerate their answers.

 Data Analysis

The degree of agreement and consensus among experts was measured by computing median scores and IQRs.

 Study 1b: Potential Future Users Round 2 Procedures and Participants

Potential future users were recruited through an independent, commercial Dutch Internet research panel [45]. From this panel, consisting of approximately 20,000 members, a sample of 220 members were invited to participate in this study. Potential users were invited to take part only in the second round of this Delphi study, since the main goal was to compare users’ opinions with experts’ opinions and not to force consensus among users on the different factors. Respondents from this panel were rewarded for their participation in this study in accordance with the standards of the consumer panel (approximately €5). All participants received an email invitation informing them about the goal and content of the study. The invitation also contained a link that directed participants to the questionnaire. A total of 210 potential future users (95.5% response rate) responded to the invitation (Table 2).

Demographic profile of potential future users (n = 210)

Demographic characteristic % n
Age (years)
Range 19–65
Mean (SD) 46.49 (47.00)
Gender
Male 51 106
Female 49 104
Ethnicity
Native 98 206
Nonnative 2 4
Education level
Low 11 24
Medium 59 123
high 30 63
Income
Less than average 29 61
Average 26 54
More than average 45 94
Internet use: work or study a
Frequent 55 116
Average 12 26
Infrequent 10 20
Not applicable 23 48
Internet use: private use a
Frequent 96 202
Average 3 6
Infrequent 1 2
Internet use: health purposes
Frequent 21 45
Average 69 144
Infrequent 10 21

a Frequent user: >3 times/week; average user: every week, but not >3 days/week; infrequent user: <1/week.

 Questionnaire

The questionnaire assessed demographics such as gender, age, ethnicity, education level, and income. In addition, participants were asked to indicate how often they used the Internet for work or study purposes (1, [almost] never; 6, every day), for private use (1, [almost] never; 6, every day), and for finding information on health-related topics (1, never; 6, very often). For the questions assessing Internet use for work and private purposes, new categories were composed: frequent user (>3 times a week), average user (every week, but not >3 days a week), and infrequent user (<1 a week).

Furthermore, the questionnaire contained all factors identified by experts in the first round of study 1a, resulting in a questionnaire consisting of 92 questions. Potential future users were asked to rate the importance of all factors on a 7-point Likert scale ranging from 1 (not at all important) to 7 (extremely important).

 Data Analysis

Data were analyzed according to the same principles used for the second round of study 1a, by calculating median scores and IQRs. Differences in consensus between the expert group and potential future users were analyzed using Wilcoxon signed rank sum tests. Similarities and differences in second-round results between experts and potential future users were further analyzed by using multivariate analysis of variance.

 Results Study 1a: Experts

We used experts’ responses from the first round to compose the questionnaire for the second round, and results of the first round are therefore shown as question items. We grouped items with regard to layout, general information content, health risk information content, ease of use, questionnaire completion, visual aids, and additional services. This resulted in a list of 85 factors thought to be related to optimal development of health-related websites. There were 10 factors in the layout category, 11 in general information content, and 16 in health risk information content. A total of 18 factors were mentioned as facilitating ease of use. The remaining factors referred to questionnaire completion (11), visual aids (6), and additional services (13). An overview of all results is given in Multimedia Appendix 1.

Consensus

During the second round, consensus was reached (IQR ≤ 1) on 57 factors. After experts rerated their answers during the third round, aided by feedback on second-round group results, consensus was obtained on another 6 factors. In total consensus was obtained on 74% of all factors.

 Importance

Experts identified 33 factors as being very or extremely important (median ≥ 6): (1) layout, with 5 factors (50% of layout-related factors, eg, professional appearance and use of color), (2) general information provision, with 4 factors (36% of general information-related factors, eg, information on pros and cons of a healthy lifestyle, and tailored information on health), (3) health risk information provision, with 3 factors (19% of health risk-related factors, eg, information on skills that help to decrease health risks and personal advice on how to decrease health risks), (4) ease of use, with 10 factors (56% of factors related to ease of use, eg, availability of an easy log-in procedure and a clear navigation structure), (5) questionnaire completion, with 6 factors (55% of factors related to completion, eg, provision of a progress bar and provision of an option for partial completion), and (6) additional services, with 5 factors (38% of factors related to additional services, eg, provision of a self-monitoring tool and iterative feedback).

Although consensus was obtained on 3 factors relating to the provision of visual aids (eg, provision of cartoons, pictures, and graphical representations of relevant information), these factors had respective median scores of 4, 5, and ,5 and thus were not considered as extremely or very important. Combining the results on importance and obtained consensus, we can conclude that consensus (IQR ≤ 1) was obtained on the importance (median ≥ 6) of 24 factors (Table 3).

Median scores for important factors in health-related Internet sites on which consensus was obtained by experts, potential users, or botha

Factor Second round Third round
Experts (n = 60) Users (n = 120) Experts (n = 32)
Median IQRb Median IQR Median IQR
Which factors determine optimal layout?
1. User friendlyc 7 1 6 2 NAd NA
2. Lively appearancec 6 1 5 1 NA NA
3. Use of visual materials, such as pictures, videos, and graphicsc 6 1 5 1 NA NA
4. Use of colorc 6 1 5 2 NA NA
5. Professional appearancee 6 2 6 1 6 1
What kind of general information should definitely be provided on health?
6. Information on how to attain a healthy lifestylec 6 1 6 1 NA NA
7. Information on pros and cons of a healthy lifestylec 6 1 5 1 NA NA
8. Information on health risk behaviorsc 6 1 5 1 NA NA
9. Personal tailored information on healthc 6 2 5 1 6 1
What kind of health risk information should definitely be provided?
10. Information in the form of visual aids, eg, graphsc 6 1 5 1 NA NA
Which factors determine optimal ease of use?
11. Clear structuref 7 1 6 1 NA NA
12. Availability of an easy log-in proceduref 7 1 6 1 NA NA
13. Use of comprehensive languagec 7 1 6 2 NA NA
14. Clear navigation structurec 7 1 2 2 NA NA
15. Simple site designf 6 1 6 1 NA NA
16. Availability of a helpdeskc 6 1 5 2 NA NA
17. Availability of a function to customize the site for personal needsc 6 1 4 2 NA NA
18. Availability of bread crumb navigatione 5 2 6 1 5 2
19. Availability of contact information for developersc 6 2 5 1 6 1
Which factors determine whether visitors complete questionnaires provided on the sites?
20. Clearly structured questionnaire with clear headings and subheadingsf 6 1 6 1 NA NA
21. Progress barf 6 1 6 1 NA NA
22. Opportunity to stop completion and proceed at a later timef 6 1 6 1 NA NA
23. Information on personal benefits of completionc 6 1 5 1 NA NA
24. Information on relevance of questionnaire completionc 6 1 5 1 NA NA
25. Guarantee that completion will not result in receiving spame 6 2 7 1 5 1
26. Use of original questionse 4 1 6 1 NA NA
What additional services should be provided?
27. Search enginee 5 1 6 1 NA NA
28. Opportunity to print or download relevant informationc 6 1 5 1 NA NA
29. Opportunity for regularly revisiting the sitec 6 1 5 1 NA NA
30. Tool to self-monitor personal health behavior changec 6 1 5 2 NA NA
31. Iterative feedback during revisits to assess users against their own previous performancesc 6 1 5 2 NA NA
32. Privacy statementc 6 2 5 2 6 1

a Only experts were included in the third round. Statements on which consensus was obtained in the second round were excluded from the third round and results are therefore missing. Only factors on which consensus was obtained, either by experts or users, or both, are displayed (interquartile range ≤ 1).

b Interquartile range.

c Factors on which consensus was obtained only by experts.

d Not applicable.

e Factors on which consensus was obtained only by potential users.

f Factors on which consensus was obtained by both experts and potential users.

 Study 1b: Potential Future Users Consensus

Potential future users reached consensus (IQR ≤ 1) on 60 of the 85 factors (71%). An overview of all results is given in Multimedia Appendix 1.

 Importance

A total of 17 factors were identified as being very or extremely important (median ≥ 6). These factors were mapped into different categories: (1) layout, with 3 factors (30% of factors related to layout, eg, limited amount of distractions and user friendliness), (2) general information provision, with 1 factor (9% of factors related to general information; information on how to attain a healthy lifestyle), (3) ease of use, with 7 factors (39% of factors related to ease of use, eg, availability of an easy log-in procedure and use of comprehensive language), (4) questionnaire completion, with 3 factors (55% of factors related to questionnaire completion, eg, provision of a progress bar and provision of an option for partial completion), and (5) additional services, with 5 factors (38% of factors related to additional services, eg, provision of a self-monitoring tool and iterative feedback).

Although consensus was obtained on 14 factors relating to ease of use (eg, information on skills that help to decrease health risks, and personal advice on how to decrease health risks), all these factors had median scores of 5 and were not considered to be extremely or very important. Furthermore, all factors related to provision of visual aids had median scores <6, even though consensus was obtained on all 6 factors. Combining the results on importance and obtained consensus, we can conclude that consensus (IQR ≤ 1) was obtained on the importance (median ≥ 6) of 11 factors (Table 3).

 Similarities and Differences Between Experts and Potential Future Users

The results of the second round revealed that experts agreed on the importance of 24 factors, whereas the majority of potential future users agreed on the importance of 11 factors (median ≥ 6; IQR ≤ 1). The statistical tests showed a nonsignificant (z = –.262, P = .29) difference between the overall level of consensus obtained in the two groups.

Experts and potential future users both agreed on the importance of 7 factors: (1) general information, with 1 factor (the availability of information on how to attain a healthy lifestyle), (2) ease of use, with 3 factors (clear structure, easy log-in procedure, simple site design), and (3) questionnaire completion, with 3 factors (provision of a progress bar, a clearly structured questionnaire, and an option for partial completion).

Contrary to these similarities in perceived importance, experts and potential future users’ opinions differed on 22 factors, significantly so on 19 factors (Table 4). The multivariate analysis of variance found a significant effect of group on the importance of factors related to optimal development (F 1,268 = 95.95, P < .001, R 2 = .90. Univariate F tests revealed that experts rated a lively appearance, the use of visual aids, the use of color, user friendliness (optimal layout), information in the form of visual aids (health risk information), opportunities for customizing the site, availability of a helpdesk, use of comprehensive language, simple site design, clear navigation structure (optimal ease of use), provision of relevance and personal benefits of completing the questionnaire (questionnaire completion), provision of a self-monitoring tool, option for printing and downloading information, stimulation of revisits, and iterative feedback (additional services) as significantly more important factors than did potential future users.

Conversely, potential future users rated the availability of bread crumb navigation (optimal ease of use), the provision of a guarantee that questionnaire completion will not result in receiving spam, and usage of original questions in questionnaires (questionnaire completion) as more important than did experts.

Univariate F tests for differences in rating the importance of a factor between experts and users

Factor Mean experts (n = 60) Mean users (n = 210) F η 2
Optimal layout
Lively appearance 5.48 (0.83) 5.01 (0.98) 11.37*** .04
Professional appearance 5.87 (0.97) 5.62 (0.81) 3.84 .01
Use of visual materials, such as pictures, videos, and graphics 5.98 (0.89) 5.21 (0.98) 30.23*** .10
Use of color 5.55 (0.89) 5.00 (1.05) 13.85*** .05
User friendly 6.68 (0.57) 6.09 (0.89) 24.08*** .08
General information
Information on pros and cons of a healthy lifestyle 5.53 (0.91) 5.36 (0.97) 1.58 .01
Information on health risk behaviors 5.60 (0.81) 5.43 (0.96) 1.59 .01
Health risk information
Information in the form of visual aids, such as graphs 5.71 (1.09) 4.88 (1.06) 28.53*** .10
Optimal ease of use
Availability of a function to customize the site for personal needs 5.78 (0.98) 3.92 (1.36) 97.63*** .27
Availability of a helpdesk 5.68 (0.89) 5.04 (1.06) 18.09*** .06
Use of comprehensive language 6.50 (0.70) 6.01 (0.91) 14.61*** .05
Availability of bread crumb navigation 5.05 (1.19) 5.63 (0.97) 15.22*** .05
Simple site design 6.02 (0.89) 5.65 (0.89) 7.88** .03
Clear navigation structure 6.57 (0.75) 2.18 (0.95) 1084.30** .80
Questionnaire completion
Information on relevance of questionnaire completion 5.77 (0.89) 5.51 (0.89) 3.92* .01
Information on personal benefits of completion 5.78 (0.92) 5.34 (0.98) 9.95** .04
Guarantee that completion will not result in receiving spam 5.87 (1.16) 6.29 (0.95) 8.44** .03
Use of original questions 4.53 (1.19) 5.59 (1.01) 46.50*** .15
Additional services
Tool to self-monitor personal health behavior change 5.88 (0.85) 4.84 (1.21) 39.41*** .13
Opportunity to print or download relevant information 6.18 (0.83) 5.41 (1.15) 23.41*** .08
Opportunity to regularly revisit the site 6.02 (0.89) 5.15 (1.13) 29.95*** .10
Iterative feedback during revisits 5.92 (0.79) 5.04 (1.07) 34.45*** .11

*P < .05, **P < .01, ***P < .001.

 Discussion

The public health impact of Internet health communication programs is suboptimal in the target population [17,18,46]. Since program development is one of the main processes to influence reach of a new product [25], this study attempted to give both experts and users a say in the development process. By means of a Delphi study, we identified an extensive list of factors potentially related to optimal development of health-related websites, as well as the extent to which experts and users agreed on the importance of these factors. In addition, we deduced similarities and differences in perceived importance between experts and potential users.

Main Findings

This study identified an extensive list of factors that might contribute to the development of health-related websites. The importance of a selected set of these factors was stressed by both experts and potential users. Developers should therefore attempt to take these factors into account when developing a health-related website. In particular, the provision of information on attaining a healthy lifestyle was emphasized. Furthermore, a clearly structured website with a simple design and the presence of an easy log-in procedure were brought up as important factors, which corresponds to results from earlier studies [31,39]. Also, with respect to optimizing questionnaire completion, several important factors were identified. To increase questionnaire completion, websites should offer a progress bar and an option for partial completion. These results are also in line with previous findings [47]. Since questionnaire length seems to be inversely related to actual participation as well as completion [48], providing an option for partial completion might be a solution to prevent attrition when using extensive questionnaires. Besides factors agreed upon by both groups, results also indicated that the majority of experts and users significantly differed on the importance of several factors. Due to their specific knowledge on theories in their area of expertise, experts should be involved in the development process to ensure scientific input. However, opinions of potential users must not be neglected because, by engaging users in this process, their involvement will increase. This involvement will subsequently increase the chances of obtaining higher levels of appreciation, prolonged and sustained use, and lower levels of attrition [37].

The recently introduced Internet intervention model [35] offers an opportunity to classify important factors identified in this study into several main areas that determine how the website is developed and functions. The first area pertains to the appearance of the website, which is one of the first website characteristics visitors are confronted with. Since previous research indicated that more than half of all website visitors are inclined to leave the website within the first 30 seconds [49], the exterior of the website should be appealing in order to attract sufficient visitor attention and prevent early disengagement. In this study, experts highlighted important factors, such as a lively appearance, the use of color, and the use of visual aids, as contributing to the appearance of health-related websites. Furthermore, potential users stressed the importance of developing websites with a professional appearance [39]. The model also emphasizes behavioral prescription, which refers to the instructions users receive on how to change their lifestyle. In the current study, both experts and potential users emphasized the importance of offering information on how to achieve a healthy lifestyle on health-related websites. Another important area of the website is the size of the burden that actual use of the website entails. Results from this study indicated that both experts and users value the presence of an easy log-in procedure. Furthermore, experts stressed the importance of developing a user-friendly website, to decrease the effort visitors must invest in navigating the website. This also entails a simple site design and a clear navigation structure. Furthermore, potential users indicated they appreciated the presence of bread crumb navigation and a search engine, to ensure visible navigation on the website [50]. C ontent that is provided on the website should be accurate, complete, and readable [2,51]. In line with these findings, experts agreed on the importance of using comprehensive language on health-related websites. Since health-related websites mainly aim at helping or assisting people to adopt a healthy lifestyle, experts stressed that detailed information on the pros and cons of a healthy lifestyle and on health behaviors in general should be included in the website’s content [31,39]. Additionally, potential users indicated the importance of an opportunity to download or print relevant information that is provided on the website. Participation is considered as another important component of the website and focuses on its ability to engage and involve visitors. Providing reinforcement is considered an important strategy to engage visitors and can be a reward for progressing through the website content [35]. The provision of personalized feedback regarding the status of lifestyle behaviors and the opportunity to self-monitor behavior change can be regarded as a form of reinforcement. In line with this, experts stressed the importance of providing a self-monitoring tool to allow visitors to assess their current health behavior status and to track their progress. Finally, assessment refers to the website’s ability to adjust the website to specific user needs and wishes. In this study, experts indicated that an opportunity to adjust the website to personal preferences was highly appreciated. Therefore, visitors should have an opportunity to adjust not only the appearance of the website but also its content, corresponding to their needs and wishes. Tailoring is considered to be an appropriate strategy to adjust health information to personal characteristics and preferences [52]. In addition, previous studies indicated that providing personalized, iterative feedback regarding one’s health behavior might in itself stimulate revisits to the program [31]. In line with these previous findings, the majority of experts agreed on the importance of stimulating revisits and using iterative feedback. To adjust information to personal characteristics and specific user needs, questionnaires are often requisite to obtain detailed information on these topics. As stressed in the introduction, these questionnaires often tend to be extensive and therefore sensitive to early dropout. It is therefore imperative to stimulate questionnaire completion in order to optimize adjustment to visitor’s needs and wishes. Experts indicated that information on relevance and personal benefit [47] of completing a questionnaire should be provided. Potential users indicated that developers should develop original questions and guarantee that questionnaire completion will not result in receiving spam.

The Internet Intervention model emphasizes two additional areas that were also identified in this Delphi study: mode of delivery and the message. Mode of delivery refers to ways in which the intervention content can be delivered to visitors. Specific strategies that can be used pertain to the use of animations, audio, video, or testimonials. The message area, on the other hand, focuses on the source and style of the message and addresses issues such as the trustworthiness and expertise of the website developers. Even though several factors pertaining to these areas were identified in the Delphi study, the majority of both experts and potential users did not agree on the importance of these factors.

 Limitations

Several limitations to this study should be considered. First, the development of health-related websites is a very broad topic entailing diverse elements. To obtain detailed information on factors related to each separate element of the development process (eg, deciding on layout, content, and additional services), the questionnaire was divided into 7 categories. This rather broad setup, which is often inherent to the Delphi process, may have limited the specification of in-depth information. As the Delphi method does not allow for further specification of such factors in later rounds, still more in-depth examination of such factors is required. Second, response rates for the expert study ranged between 33% and 53%. Although this range is somewhat low, previous Delphi studies have reported similar response rates [31,43,53]. Suboptimal response rates in our study might have been a consequence of experts being invited to participate in at least two rounds of the Delphi study. Especially, response rates among experts with a practice-based background, coming from the field of technical Web design, were low during the first round. To account for these suboptimal response rates and to balance the input from the various expert fields, we put additional effort into recruiting these experts for participation in the second round. Finally, to ensure dispersion and coverage of answers, as well as optimal participation rates, it is recommendable for future studies to limit the number of questions used in the Delphi study. Incorporating a small number of questions in the first-round questionnaire may allow experts to give more in-depth input and may increase participation in subsequent rounds, thereby diminishing attrition rates.

 Implications

The currents study is a first step in increasing exposure rates of online health promotion programs. The results of this study need further experimental testing to identify which (combination of) factors ultimately results in the best result. Although the vast number of factors that play a role may hinder the feasibility of a full factorial experimental design, in-depth experimental studies on the importance of some categories of factors are recommended.

 Multimedia Appendix 1

Results of the Delphi study per factor for experts and potential future users (second and third round).

 Abbreviations IQR

interquartile range

This study was funded by ZonMw, the Netherlands Organization for Health Research and Development (grant number: 120710004). Publication of the manuscript was supported by NWO, the Netherlands Organization for Scientific Research. The authors would like to thank Kim Eijmael and Claire Dolmans for their assistance during the recruitment phases and analysis of the first-round results.

Hein de Vries is scientific director of Vision2Health, a company that licenses evidence-based innovative computer-tailored health communication tools.

 Bansil P Keenan NL Zlot AI Gilliland JC Health-related information on the Web: results from the HealthStyles Survey, 2002-2003 Prev Chronic Dis 2006 04 3 2 A36 16539777 A36 PMC1563974 Eysenbach G Kohler Ch What is the prevalence of health-related searches on the World Wide Web? Qualitative and quantitative analysis of search engine queries on the internet AMIA Annu Symp Proc 2003 225 9 14728167 D030003690 PMC1480194 Internet World Stats Miniwatts Marketing Group 2011 10 6 2011-11-25 Internet Usage Statistics: The Internet Big Picturehttp://www.internetworldstats.com/stats.htm 63SbuP6d0 Brug J Oenema A Kroeze W Raat H The internet and nutrition education: challenges and opportunities Eur J Clin Nutr 2005 08 59 Suppl 1 S130 7; discussion S138 10.1038/sj.ejcn.1602186 16052181 1602186 Ritterband LM Gonder-Frederick LA Cox DJ Clifton AD West RW Borowitz S Internet interventions: in review, in use and into the future Prof Psychol Res Pr 2003 34 527 34 10.1037/0735-7028.34.5.527 Krebs P Prochaska JO Rossi JS A meta-analysis of computer-tailored interventions for health behavior change Prev Med 2010 51 3-4 214 21 10.1016/j.ypmed.2010.06.004 20558196 S0091-7435(10)00231-8 PMC2939185 Vandelanotte C Spathonis KM Eakin EG Owen N Website-delivered physical activity interventions a review of the literature Am J Prev Med 2007 07 33 1 54 64 10.1016/j.amepre.2007.02.041 17572313 S0749-3797(07)00164-X van den Berg MH Schoones JW Vliet Vlieland TP Internet-based physical activity interventions: a systematic review of the literature J Med Internet Res 2007 9 3 e26 10.2196/jmir.9.3.e26 17942388 v9i3e26 PMC2047289 Neville LM O'Hara B Milat AJ Computer-tailored dietary behaviour change interventions: a systematic review Health Educ Res 2009 08 24 4 699 720 10.1093/her/cyp006 19286893 cyp006 PMC2706490 Brug J Glanz K Van Assema P Kok G van Breukelen GJ The impact of computer-tailored feedback and iterative feedback on fat, fruit, and vegetable intake Health Educ Behav 1998 08 25 4 517 31 9690108 Strecher VJ Computer-tailored smoking cessation materials: a review and discussion Patient Educ Couns 1999 02 36 2 107 17 10223016 S0738-3991(98)00128-1 Myung SK McDonnell DD Kazinets G Seo HG Moskowitz JM Effects of Web- and computer-based smoking cessation programs: meta-analysis of randomized controlled trials Arch Intern Med 2009 05 25 169 10 929 37 10.1001/archinternmed.2009.109 19468084 169/10/929 Borland R Balmford J Hunt D The effectiveness of personally tailored computer-generated advice letters for smoking cessation Addiction 2004 03 99 3 369 77 10.1111/j.1360-0443.2003.00623.x 14982550 ADD623 Dijkstra A De Vries H Roijackers J Long-term effectiveness of computer-generated tailored feedback in smoking cessation Health Educ Res 1998 06 13 2 207 14 10181019 Riper H Kramer J Smit F Conijn B Schippers G Cuijpers P Web-based self-help for problem drinkers: a pragmatic randomized trial Addiction 2008 02 103 2 218 27 10.1111/j.1360-0443.2007.02063.x 18199300 ADD2063 White A Kavanagh D Stallman H Klein B Kay-Lambkin F Proudfoot J Drennan J Connor J Baker A Hines E Young R Online alcohol interventions: a systematic review J Med Internet Res 2010 12 5 e62 10.2196/jmir.1479 21169175 v12i5e62 PMC3057310 Eysenbach G The law of attrition J Med Internet Res 2005 7 1 e11 10.2196/jmir.7.1.e11 15829473 v7e11 PMC1550631 De Nooijer JOA Kloek G Brug J De Vries H De Vries NK Bevordering van gezondheidsgedrag via internet: nu en in de toekomst [Promotion of Healthy Behavior Through the Internet: Now and in the Future] 2005 Maastricht Maastricht University Crutzen R Hard to Get, Hard to Keep: Dissemination of and Exposure to Internet-Delivered Health Behavior Change Interventions Aimed at Adolescents [dissertation] 2009 Maastricht Maastricht University Glasgow RE Nelson CC Kearney KA Reid R Ritzwoller DP Strecher VJ Couper MP Green B Wildenhaus K Reach, engagement, and retention in an Internet-based weight loss program in a multi-site randomized controlled trial J Med Internet Res 2007 9 2 e11 10.2196/jmir.9.2.e11 17513282 v9i2e11 PMC1874718 Evers KE Cummins CO Prochaska JO Prochaska JM Online health behavior and disease management programs: are we ready for them? Are they ready for us? J Med Internet Res 2005 07 1 7 3 e27 10.2196/jmir.7.3.e27 15998618 v7i3e27 PMC1550656 Cook C Heath F Thompson RL A meta-analysis of response rates in web- or Internet-based surveys Educ Psychol Meas 2000 60 821 36 10.1177/00131640021970934 de Vries H Brug J Computer-tailored interventions motivating people to adopt health promoting behaviours: introduction to a new approach Patient Educ Couns 1999 02 36 2 99 105 10223015 S073839919800127X Glasgow RE Vogt TM Boles SM Evaluating the public health impact of health promotion interventions: the RE-AIM framework Am J Public Health 1999 09 89 9 1322 7 10474547 PMC1508772 Rogers E Diffusion of Innovations 2003 New York, NY Free Press Danaher BG Boles SM Akers L Gordon JS Severson HH Defining participant exposure measures in Web-based health behavior change programs J Med Internet Res 2006 8 3 e15 10.2196/jmir.8.3.e15 16954125 v8i3e15 PMC1761946 Crutzen R de Nooijer J Candel MJ de Vries NK Adolescents who intend to change multiple health behaviours choose greater exposure to an internet-delivered intervention J Health Psychol 2008 10 13 7 906 11 10.1177/1359105308095064 18809641 13/7/906 Verheijden MW Jans MP Hildebrandt VH Hopman-Rock M Rates and determinants of repeated participation in a web-based behavior change program for healthy body weight and healthy lifestyle J Med Internet Res 2007 9 1 e1 10.2196/jmir.9.1.e1 17478410 v9i1e1 PMC1794672 Christensen H Griffiths KM Farrer L Adherence in internet interventions for anxiety and depression J Med Internet Res 2009 11 2 e13 10.2196/jmir.1194 19403466 v11i2e13 PMC2762797 Norman GJ Zabinski MF Adams MA Rosenberg DE Yaroch AL Atienza AA A review of eHealth interventions for physical activity and dietary behavior change Am J Prev Med 2007 10 33 4 336 45 10.1016/j.amepre.2007.05.007 17888860 S0749-3797(07)00363-7 PMC2180189 Brouwer W Oenema A Crutzen R de Nooijer J de Vries NK Brug J An exploration of factors related to dissemination of and exposure to internet-delivered behavior change interventions aimed at adults: a Delphi study approach J Med Internet Res 2008 10 2 e10 10.2196/jmir.956 18417443 v10i2e10 PMC2483924 Moorman C Matulich E A model of consumers' preventive health behaviors: the role of health motivation and health ability J Consum Res 1993 20 208 28 Brouwer W Kroeze W Crutzen R de Nooijer J de Vries NK Brug J Oenema A Which intervention characteristics are related to more exposure to internet-delivered healthy lifestyle promotion interventions? A systematic review J Med Internet Res 2011 13 1 e2 10.2196/jmir.1639 21212045 v13i1e2 PMC3221341 Fry JP Neff RA Periodic prompts and reminders in health promotion and health behavior interventions: systematic review J Med Internet Res 2009 11 2 e16 10.2196/jmir.1138 19632970 v11i2e16 PMC2762806 Ritterband LM Thorndike FP Cox DJ Kovatchev BP Gonder-Frederick LA A behavior change model for internet interventions Ann Behav Med 2009 08 38 1 18 27 10.1007/s12160-009-9133-4 19802647 PMC2878721 Cugelman B Thelwall M Dawes P Online interventions for social marketing health behavior change campaigns: a meta-analysis of psychological architectures and adherence factors J Med Internet Res 2011 13 1 e17 10.2196/jmir.1367 21320854 v13i1e17 PMC3221338 Glasgow RE Klesges LM Dzewaltowski DA Bull SS Estabrooks P The future of health behavior change research: what is needed to improve translation of research into health promotion practice? Ann Behav Med 2004 02 27 1 3 12 10.1207/s15324796abm2701_2 14979858 Cunningham M Rapee R Lyneham H Feedback to a prototype self-help computer program for anxiety disorders in adolescents Adv Ment Health 2006 5 3 216 24 Eysenbach G Köhler C How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews BMJ 2002 03 9 324 7337 573 7 11884321 PMC78994 Adler M Ziglio E Gazing Into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health 1996 Bristol, PA Jessica Kingsley Publishers Crutzen R de Nooijer J Brouwer W Oenema A Brug J de Vries NK Internet-delivered interventions aimed at adolescents: a Delphi study on dissemination and exposure Health Educ Res 2008 06 23 3 427 39 10.1093/her/cym094 18209115 cym094 Elfeddali I Bolman C Mesters I Wiers RW de Vries H Factors underlying smoking relapse prevention: results of an international Delphi study Health Educ Res 2010 12 25 6 1008 20 10.1093/her/cyq053 20864604 cyq053 De Vet E Brug J De Nooijer J Dijkstra A De Vries NK Determinants of forward stage transitions: a Delphi study Health Educ Res 2005 04 20 2 195 205 10.1093/her/cyg111 15328302 cyg111 Linstone HA Turoff M Introduction to the Delphi method: techniques and applications Linstone HA, Turoff M, editors. The Delphi Method: Techniques and Applications 1975 Reading, MA Addison-Wesley 3 12 FlyCatcher Internet Research 2011-11-25 Homepagehttp://www.flycatcher.eu/ 63Sbdxa2b Glasgow RE eHealth evaluation and dissemination research Am J Prev Med 2007 05 32 5 Suppl S119 26 10.1016/j.amepre.2007.01.023 17466816 S0749-3797(07)00052-9 Crutzen R Optimalisering van de betrokkenheid van deelnemers en datakwaliteit in een cohortstudie [Optimizing Participants' Involvement and Data Quality in a Cohort Study] 2010 Maastricht Maastricht University Galesic M Bosjnak M Effects of questionnaire length on participation and indicators of response quality in a Web survey Public Opin Q 2009 73 349 60 Brouwer W Oenema A Raat H Crutzen R de Nooijer J de Vries NK Brug J Characteristics of visitors and revisitors to an Internet-delivered computer-tailored lifestyle intervention implemented for use by the general public Health Educ Res 2010 08 25 4 585 95 10.1093/her/cyp063 19897515 cyp063 Bognazzini B Nielsen Norman Group 2011-11-25 Ask TOG: First Principles of Interaction Designhttp://www.asktog.com/basics/firstPrinciples.html 63ScwBERH Eysenbach G Powell J Kuss O Sa ER Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review JAMA 2002 287 20 2691 700 12020305 jrv10005 Kreuter M Farrell D Olevitch L Brennan L Tailoring Health Messages: Customizing Communication with Computer Technology 2000 Mahwah, NJ Lawrence Erlbaum Associates Mullen PM Delphi: myths and reality J Health Organ Manag 2003 17 1 37 52 12800279