Every 10 years, ODPHP manages the development of a set of measurable objectives for improving the health of the nation over the coming decade, known as Healthy People. Two objectives (Health Communication and Health Information Technology, HC/HIT-8.1 and 8.2, see Textbox 1) in Healthy People 2020 relate to website quality: one calls for improved information reliability; the other calls for improved website usability. Both were devised and operationalized by experts in health communication and technology. Together, these objectives provide a working definition of website quality and a promising path toward overall quality improvement of Web-based health information.

Information reliability refers to the accuracy and credibility of website content as well as transparency in the purpose and ownership of the site [7,19]. This information can help users discern the origin and quality of Web-based content [20]. It is one of the most commonly identified indicators of website quality and has been widely referenced by public, private, and nonprofit organizations committed to improving the quality of Web-based health information such as the Medical Library Association, Health on the Net Foundation, Consumers Union, and the Agency for Healthcare Research and Quality [21-24].

Healthy People objective HC/HIT-8.1 was first introduced in Healthy People 2010 as a developmental objective. In 2005, ODPHP convened a Technical Expert Workgroup to identify reliability criteria based on established Web standards [25,26]. The ODPHP then developed the Website Information Reliability Evaluation Instrument (Multimedia Appendix 1), which includes specific reliability requirements for each of the 6 criteria identified by the Workgroup (see Table 1).

Usability standards tend to fall into three major categories [27]. The first category focuses on how the information is organized, commonly referred to as information architecture. The second category looks at how users navigate the information on a website, known as site design. The third category emphasizes how users interact with content on the website, referred to as content design. These three elements are commonly addressed in both federal and private guidelines on website usability [27-32].

Usability is an important component of website quality, affecting a user’s ability to access and understand information online [33]. In fact, the design of a website is one of the most important indicators of website credibility and quality for users [34-37].

In 2012, an expert panel was used to establish an empirical definition of usability for Healthy People 2020. Panel members were selected from academic, private, and government sectors based on their expertise in website usability and health communication. With input from the panel, ODPHP developed the Website Usability Evaluation Instrument (see Multimedia Appendix 2) to measure progress toward Healthy People objective HC/HIT-8.2. The instrument assesses the three aforementioned website usability categories, using 19 established usability principles across 59 task-based usability measures (see Table 2). The Site Design category includes 9 composites that assess basic design elements of the site, including how the site looks, how the site functions, and how a user can interact with the site. The Information Architecture category includes 7 composites that assess how the site content is organized, including navigation, grouping, and labeling. Lastly, the Content Design category includes 3 composites that assess how the content is written and formatted, and includes plain language principles. Each of the 59 measures is rated on a scale of 1 to 4 based on the level of difficulty of performing the task on the website (1 being “task failure” and 4 being “minimal problems”). An average rating score is calculated for each usability principle. The benchmark was set to require an average score of 3.5 for 10 or more of the 19 usability principles.

With Healthy People objectives HC/HIT-8.1 and 8.2, ODPHP established standardized criteria to define and measure the quality of health-related websites and set benchmarks for measurement. Next, the criteria were applied to a sample of health-related websites in order to identify baseline data points, which are required for all Healthy People measurable objectives. The reliability (objective HC/HIT-8.1) and usability (objective HC/HIT-8.2) instruments were combined into a single instrument: the National Quality Health Website Survey. Targets were set to drive improvement by the year 2020.

The ODPHP identified the 100 top-ranked health-related websites (see Multimedia Appendix 3) for 3 months (August-October 2014) from the Alexa Top Sites pool—health category [38]. The data were collected from Alexa on October 14, 2014. Alexa’s traffic ranks are based on the traffic data provided by users in Alexa’s global data panel over a rolling 3-month period. A site’s ranking is based on a combined measure of unique visitors and page views [39]. Websites were considered health related if they had at least three items of health information as it is broadly defined by the e-Health Code of Ethics (see Textbox 2) [40]. Duplicated websites were consolidated with the exception of microsites within the National Institutes of Health such as PubMed and MedlinePlus. A number of websites were excluded from the sample based on the exclusion criteria in Textbox 3.

The final sample included websites sponsored by three types of organizations: 48 out of 100 websites (48.0%) were for profit, 36 out of 100 (36.0%) were nonprofit, and 16 out of 100 (16.0%) were government.

A senior usability researcher and a research associate reviewed the sample websites, following a reviewer training process to reach a certain level of interrater reliability (IRR). The team used Altman’s Kappa Benchmark Scale (see Table 3) [41].

Objective HC/HIT-8.1 criteria are primarily composed of yes or no questions. Cohen’s kappa was used to measure the agreement for nominal scales. A benchmark kappa score of .80 (a score generally accepted as demonstrating a sufficient degree of IRR) was used for objective HC/HIT-8.1 [41]. The criteria for objective HC/HIT-8.2 are scored on a 4-point scale, making it more difficult to reach a perfect IRR. A benchmark kappa score of .61 was used for objective HC/HIT-8.2. Interclass correlation (ICC) was used for assessing ordinal and interval scales. To ensure IRR, both reviewers assessed the same 6 websites during the initial training process. The IRR was calculated for each assessment; discrepancies were identified and resolved between the 2 reviewers until the IRR scores met the benchmark kappa scores.

After the initial training process, the reviewers randomly selected 4 additional websites from the sample to measure IRR scores. The IRR scores for the Website Information Reliability Evaluation Instrument (kappa .83) and the Website Usability Evaluation Instrument (ICC .76) both met the benchmarks.

After ensuring IRR, the reviewers divided the remaining 90 websites into 2 equal groups. Each reviewer assessed an equal number of websites.

To score each item in the Website Information Reliability Evaluation Instrument, reviewers began by randomly selecting 3 pages from different sections of the site. Each of the items in the instrument was scored based on a review of at least 3 different pages and no individual page was reviewed more than once. For items in the instrument that refer to a specific page or feature (eg, home page–related items and search function items), the review also included those specific pages or features. As each review progressed, the reviewer revised the scores for previously scored items as needed. In general, reviewers examined about 150 pages on each website.

For Healthy People objective HC/HIT-8.1, a total of 58 out of 100 health-related websites (58.0%) met 3 or more of the 6 information reliability criteria. Only 2 out of 100 websites (2.0%) met all the criteria, and 1 out of 100 websites (1.0%) met none of the 6 criteria. See Figure 1.

Figure 2 shows the percentage of health-related websites in compliance with specific criteria. User Feedback (90.0%) had the highest percentage in compliance, followed by Privacy (83.0%). Only 4 out of 100 websites (4.0%) met the criterion for Content Updating.

Table 4 presents the percentage of websites in compliance by criterion and by information reliability requirements associated with each criterion.

For Healthy People objective HC/HIT-8.2, a total of 42 out of 100 health-related websites (42.0%) met 10 or more out of 19 established usability principles. Figure 3 shows the distribution of website compliance by the number of usability principles met. All websites met at least one usability principle. Almost half of the websites (47.0%) met between 9 and 11 principles. None of the websites met 15 or more of the established usability principles. See Table 5 for the percentage of websites in compliance broken down by the 19 usability principles.

The 2015 review of health-related websites identified 2 baseline data points. For Healthy People 2020 objective HC/HIT-8.1, a total of 58 out of 100 health-related websites (58.0%) met 3 or more out of 6 reliability criteria. For Healthy People 2020 objective HC/HIT-8.2, a total of 42 out of 100 health-related websites (42.0%) followed 10 or more out of 19 established usability principles.

This research revealed significant shortcomings in the quality of today’s Web-based health information landscape, particularly in disclosing sources of funding and authorship, clearly differentiating between advertisements and original content, complying with universal accessibility guidelines, providing simple search and print functionality, and minimizing scientific and technical jargon.

The ODPHP set the following targets for 2020 that meet the minimal statistical significance. For Healthy People 2020 objective HC/HIT-8.1, 70.5% of health-related websites will meet 3 or more out of 6 reliability criteria. For Healthy People 2020 objective HC/HIT-8.2, 55.7% of health-related websites will follow 10 or more out of 19 established usability principles.

Improving the quality of health-related websites is critical to national efforts to promote health literacy and shared decision making. Until now, there has been no standardized approach to defining and measuring the quality of Web-based health information. With Healthy People 2020 objectives HC/HIT-8.1 and 8.2, ODPHP has developed and validated such an approach, and established baseline data points and national benchmarks to track progress over time.

The 2015 study confirmed that there is, indeed, room for improvement. The ODPHP’s research revealed significant shortcomings in the quality of today’s Web-based health information landscape, particularly in the following areas:

This study has several limitations. First, the samples of this study were selected from Alexa Top Sites—health category. The research team had no control over the quality of the website rankings performed by Alexa.

Additionally, some of the usability principles might change over time. For example, in this study, only 24 out of 100 websites (24.0%) followed the principle of minimizing vertical scrolling. However, with the proliferation of mobile and responsive design technology, users are becoming more accustomed to navigating websites by scrolling. Minimizing vertical scrolling may not remain a usability principle in the future.

Finally, the survey instrument is somewhat subjective, especially for objective HC/HIT-8.2. Although the research team controlled the reliability by measuring the IRR for several websites in the sample, there was still variation across reviewers that may affect the results.

The quality and accessibility of Web-based health information is a key factor in improving access to health services and facilitating informed health decision making. The criteria proposed by ODPHP provide methods to assess the quality of health-related websites and provide baseline data and targets to drive quality improvement. In addition to having implications for website developers and policy makers, this work also points to the need for consumer education related to the quality of Web-based health information. Such education efforts are critical in a time when nearly 3 in 4 Internet users are looking for health information online [42].

To promote increased quality of health-related websites, ODPHP updated and published Health Literacy Online: A Guide to Simplifying the User Experience, second edition [43]. Health Literacy Online is based on literature related to cognitive processing and online behavior and on usability research with more than 800 participants. It features actionable information that website owners, content writers, designers, and developers can use to create quality health websites. The recommendations in Health Literacy Online provide a clear road map for achieving the Healthy People 2020 objectives to increase the proportion of quality health-related websites (Objective HC/HIT-8). The ODPHP is working with other federal agencies to adopt the principles in Health Literacy Online and is presenting these strategies for improving health websites at national conferences.