JMIR Publications

Journal of Medical Internet Research

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 02.08.16 in Vol 18, No 8 (2016): August

This paper is in the following e-collection/theme issue:

    Original Paper

    Making Quality Health Websites a National Public Health Priority: Toward Quality Standards

    1U.S. Department of Health and Human Services (HHS), Office of Disease Prevention and Promotion (ODPHP), Rockville, MD, United States

    2U.S. Department of Health and Human Services (HHS), National Vaccine Program Office, Washington, DC, United States

    3CommunicateHealth, Inc., Northampton, MA, United States

    *all authors contributed equally

    Corresponding Author:

    Huijuan Wu, MSc, MEng, PhD

    CommunicateHealth, Inc.

    26 Market St.

    Northampton, MA, 01060

    United States

    Phone: 1 413 582 0425

    Fax:1 413 517 0545

    Email:


    ABSTRACT

    Background: Most US adults have limited health literacy skills. They struggle to understand complex health information and services and to make informed health decisions. The Internet has quickly become one of the most popular places for people to search for information about their health, thereby making access to quality information on the Web a priority. However, there are no standardized criteria for evaluating Web-based health information. Every 10 years, the US Department of Health and Human Services' Office of Disease Prevention and Health Promotion (ODPHP) develops a set of measurable objectives for improving the health of the nation over the coming decade, known as Healthy People. There are two objectives in Healthy People 2020 related to website quality. The first is objective Health Communication and Health Information Technology (HC/HIT) 8.1: increase the proportion of health-related websites that meet 3 or more evaluation criteria for disclosing information that can be used to assess information reliability. The second is objective HC/HIT-8.2: increase the proportion of health-related websites that follow established usability principles.

    Objective: The ODPHP conducted a nationwide assessment of the quality of Web-based health information using the Healthy People 2020 objectives. The ODPHP aimed to establish (1) a standardized approach to defining and measuring the quality of health websites; (2) benchmarks for measurement; (3) baseline data points to capture the current status of website quality; and (4) targets to drive improvement.

    Methods: The ODPHP developed the National Quality Health Website Survey instrument to assess the quality of health-related websites. The ODPHP used this survey to review 100 top-ranked health-related websites in order to set baseline data points for these two objectives. The ODPHP then set targets to drive improvement by 2020.

    Results: This study reviewed 100 health-related websites. For objective HC/HIT-8.1, a total of 58 out of 100 (58.0%) websites met 3 or more out of 6 reliability criteria. For objective HC/HIT-8.2, a total of 42 out of 100 (42.0%) websites followed 10 or more out of 19 established usability principles. On the basis of these baseline data points, ODPHP set targets for the year 2020 that meet the minimal statistical significance—increasing objective HC/HIT-8.1 data point to 70.5% and objective HC/HIT-8.2 data point to 55.7%.

    Conclusions: This research is a critical first step in evaluating the quality of Web-based health information. The criteria proposed by ODPHP provide methods to assess website quality for professionals designing, developing, and managing health-related websites. The criteria, baseline data, and targets are valuable tools for driving quality improvement.

    J Med Internet Res 2016;18(8):e211

    doi:10.2196/jmir.5999

    KEYWORDS



    Introduction

    Most US adults (90%) have limited health literacy skills [1], which means that many struggle to make sense of the complex information and services intended to help prevent disease and promote our health [2-5]. One promising path to improve the health literacy of the adult population is to increase the availability of evidence-based, understandable, easy-to-find sources of health information [6-8].

    While medical experts continue to play a vital role in the health decision–making process, the Internet has quickly become one of the most popular places for people to search for information about their health [9,10]. Research indicates that not only are adults of all generations searching online for health information [11,12], but they are using what they find to make health care decisions, either on behalf of themselves or a loved one [13]. Thus, improving the quality of health-related websites has the potential to improve the health literacy—and the health—of the population [14,15].

    Increasing access to quality online information is a shared priority among national and international policy makers. President Obama’s Digital Government Strategy calls for new and better ways to deliver digital information and services [16] and the World Health Organization has called for greater transparency, privacy and security, codes of conduct, and individual choice and control of health-related websites [17].

    Yet, in spite of the critical role of quality in health-related websites, there are no standardized criteria for assessing it [6,8,18]. Such criteria have been elusive since researchers called for operationalized definitions of quality in their 2002 meta-analysis of empirical website assessments [7]. Therefore, the first steps in achieving real gains in health-related website quality are to establish (1) a standardized approach to defining and measuring the quality of health websites; (2) benchmarks for measurement; (3) baseline data points to capture the current status of website quality; and (4) targets to drive improvement. This paper describes the efforts of the US Department of Health and Human Services’ Office of Disease Prevention and Health Promotion (ODPHP) to establish these four items.


    Methods

    Defining and Measuring Quality Health Websites for Healthy People

    Every 10 years, ODPHP manages the development of a set of measurable objectives for improving the health of the nation over the coming decade, known as Healthy People. Two objectives (Health Communication and Health Information Technology, HC/HIT-8.1 and 8.2, see Textbox 1) in Healthy People 2020 relate to website quality: one calls for improved information reliability; the other calls for improved website usability. Both were devised and operationalized by experts in health communication and technology. Together, these objectives provide a working definition of website quality and a promising path toward overall quality improvement of Web-based health information.


    Textbox 1. Healthy People objectives Health Communication and Health Information Technology (HC/HIT) 8.1 and 8.2.
    View this box

    Information Reliability

    Information reliability refers to the accuracy and credibility of website content as well as transparency in the purpose and ownership of the site [7,19]. This information can help users discern the origin and quality of Web-based content [20]. It is one of the most commonly identified indicators of website quality and has been widely referenced by public, private, and nonprofit organizations committed to improving the quality of Web-based health information such as the Medical Library Association, Health on the Net Foundation, Consumers Union, and the Agency for Healthcare Research and Quality [21-24].

    Healthy People objective HC/HIT-8.1 was first introduced in Healthy People 2010 as a developmental objective. In 2005, ODPHP convened a Technical Expert Workgroup to identify reliability criteria based on established Web standards [25,26]. The ODPHP then developed the Website Information Reliability Evaluation Instrument (Multimedia Appendix 1), which includes specific reliability requirements for each of the 6 criteria identified by the Workgroup (see Table 1).

    Table 1. Objective HC/HIT-8.1 criteria and reliability requirements.
    View this table

    Website Usability

    Usability standards tend to fall into three major categories [27]. The first category focuses on how the information is organized, commonly referred to as information architecture. The second category looks at how users navigate the information on a website, known as site design. The third category emphasizes how users interact with content on the website, referred to as content design. These three elements are commonly addressed in both federal and private guidelines on website usability [27-32].

    Usability is an important component of website quality, affecting a user’s ability to access and understand information online [33]. In fact, the design of a website is one of the most important indicators of website credibility and quality for users [34-37].

    In 2012, an expert panel was used to establish an empirical definition of usability for Healthy People 2020. Panel members were selected from academic, private, and government sectors based on their expertise in website usability and health communication. With input from the panel, ODPHP developed the Website Usability Evaluation Instrument (see Multimedia Appendix 2) to measure progress toward Healthy People objective HC/HIT-8.2. The instrument assesses the three aforementioned website usability categories, using 19 established usability principles across 59 task-based usability measures (see Table 2). The Site Design category includes 9 composites that assess basic design elements of the site, including how the site looks, how the site functions, and how a user can interact with the site. The Information Architecture category includes 7 composites that assess how the site content is organized, including navigation, grouping, and labeling. Lastly, the Content Design category includes 3 composites that assess how the content is written and formatted, and includes plain language principles. Each of the 59 measures is rated on a scale of 1 to 4 based on the level of difficulty of performing the task on the website (1 being “task failure” and 4 being “minimal problems”). An average rating score is calculated for each usability principle. The benchmark was set to require an average score of 3.5 for 10 or more of the 19 usability principles.

    Table 2. Objective HC/HIT-8.2 established usability principles and measures.
    View this table

    With Healthy People objectives HC/HIT-8.1 and 8.2, ODPHP established standardized criteria to define and measure the quality of health-related websites and set benchmarks for measurement. Next, the criteria were applied to a sample of health-related websites in order to identify baseline data points, which are required for all Healthy People measurable objectives. The reliability (objective HC/HIT-8.1) and usability (objective HC/HIT-8.2) instruments were combined into a single instrument: the National Quality Health Website Survey. Targets were set to drive improvement by the year 2020.

    Sampling

    The ODPHP identified the 100 top-ranked health-related websites (see Multimedia Appendix 3) for 3 months (August-October 2014) from the Alexa Top Sites pool—health category [38]. The data were collected from Alexa on October 14, 2014. Alexa’s traffic ranks are based on the traffic data provided by users in Alexa’s global data panel over a rolling 3-month period. A site’s ranking is based on a combined measure of unique visitors and page views [39]. Websites were considered health related if they had at least three items of health information as it is broadly defined by the e-Health Code of Ethics (see Textbox 2) [40]. Duplicated websites were consolidated with the exception of microsites within the National Institutes of Health such as PubMed and MedlinePlus. A number of websites were excluded from the sample based on the exclusion criteria in Textbox 3.


    Textbox 2. Criteria for selecting health websites (from e-Health Code of Ethics).
    View this box

    Textbox 3. Exclusion criteria.
    View this box

    The final sample included websites sponsored by three types of organizations: 48 out of 100 websites (48.0%) were for profit, 36 out of 100 (36.0%) were nonprofit, and 16 out of 100 (16.0%) were government.

    Interrater Reliability

    A senior usability researcher and a research associate reviewed the sample websites, following a reviewer training process to reach a certain level of interrater reliability (IRR). The team used Altman’s Kappa Benchmark Scale (see Table 3) [41].

    Table 3. Altman’s Kappa Benchmark Scale.
    View this table

    Objective HC/HIT-8.1 criteria are primarily composed of yes or no questions. Cohen’s kappa was used to measure the agreement for nominal scales. A benchmark kappa score of .80 (a score generally accepted as demonstrating a sufficient degree of IRR) was used for objective HC/HIT-8.1 [41]. The criteria for objective HC/HIT-8.2 are scored on a 4-point scale, making it more difficult to reach a perfect IRR. A benchmark kappa score of .61 was used for objective HC/HIT-8.2. Interclass correlation (ICC) was used for assessing ordinal and interval scales. To ensure IRR, both reviewers assessed the same 6 websites during the initial training process. The IRR was calculated for each assessment; discrepancies were identified and resolved between the 2 reviewers until the IRR scores met the benchmark kappa scores.

    After the initial training process, the reviewers randomly selected 4 additional websites from the sample to measure IRR scores. The IRR scores for the Website Information Reliability Evaluation Instrument (kappa .83) and the Website Usability Evaluation Instrument (ICC .76) both met the benchmarks.

    Website Review Process

    After ensuring IRR, the reviewers divided the remaining 90 websites into 2 equal groups. Each reviewer assessed an equal number of websites.

    To score each item in the Website Information Reliability Evaluation Instrument, reviewers began by randomly selecting 3 pages from different sections of the site. Each of the items in the instrument was scored based on a review of at least 3 different pages and no individual page was reviewed more than once. For items in the instrument that refer to a specific page or feature (eg, home page–related items and search function items), the review also included those specific pages or features. As each review progressed, the reviewer revised the scores for previously scored items as needed. In general, reviewers examined about 150 pages on each website.


    Results

    Baseline data points for 2015 were calculated for both Healthy People 2020 website quality objectives.

    Information Reliability Baseline

    For Healthy People objective HC/HIT-8.1, a total of 58 out of 100 health-related websites (58.0%) met 3 or more of the 6 information reliability criteria. Only 2 out of 100 websites (2.0%) met all the criteria, and 1 out of 100 websites (1.0%) met none of the 6 criteria. See Figure 1.

    Figure 2 shows the percentage of health-related websites in compliance with specific criteria. User Feedback (90.0%) had the highest percentage in compliance, followed by Privacy (83.0%). Only 4 out of 100 websites (4.0%) met the criterion for Content Updating.

    Table 4 presents the percentage of websites in compliance by criterion and by information reliability requirements associated with each criterion.

    Table 4. Estimated percentages of websites in compliance, by information reliability criterion and required disclosure elements.
    View this table
    Figure 1. Percentage of websites in compliance, with the number of reliability criteria.
    View this figure
    Figure 2. Percentage of websites in compliance, by specific reliability criteria.
    View this figure
    Table 5. Estimated percentages of websites in compliance, by criterion and established usability principles.
    View this table
    Figure 3. Percentage of websites in compliance, by the number of usability principles.
    View this figure

    Website Usability Baseline

    For Healthy People objective HC/HIT-8.2, a total of 42 out of 100 health-related websites (42.0%) met 10 or more out of 19 established usability principles. Figure 3 shows the distribution of website compliance by the number of usability principles met. All websites met at least one usability principle. Almost half of the websites (47.0%) met between 9 and 11 principles. None of the websites met 15 or more of the established usability principles. See Table 5 for the percentage of websites in compliance broken down by the 19 usability principles.


    Discussion

    Baseline Data Points

    The 2015 review of health-related websites identified 2 baseline data points. For Healthy People 2020 objective HC/HIT-8.1, a total of 58 out of 100 health-related websites (58.0%) met 3 or more out of 6 reliability criteria. For Healthy People 2020 objective HC/HIT-8.2, a total of 42 out of 100 health-related websites (42.0%) followed 10 or more out of 19 established usability principles.

    This research revealed significant shortcomings in the quality of today’s Web-based health information landscape, particularly in disclosing sources of funding and authorship, clearly differentiating between advertisements and original content, complying with universal accessibility guidelines, providing simple search and print functionality, and minimizing scientific and technical jargon.

    2020 Targets

    The ODPHP set the following targets for 2020 that meet the minimal statistical significance. For Healthy People 2020 objective HC/HIT-8.1, 70.5% of health-related websites will meet 3 or more out of 6 reliability criteria. For Healthy People 2020 objective HC/HIT-8.2, 55.7% of health-related websites will follow 10 or more out of 19 established usability principles.

    Improving the quality of health-related websites is critical to national efforts to promote health literacy and shared decision making. Until now, there has been no standardized approach to defining and measuring the quality of Web-based health information. With Healthy People 2020 objectives HC/HIT-8.1 and 8.2, ODPHP has developed and validated such an approach, and established baseline data points and national benchmarks to track progress over time.

    The 2015 study confirmed that there is, indeed, room for improvement. The ODPHP’s research revealed significant shortcomings in the quality of today’s Web-based health information landscape, particularly in the following areas:

    • Disclosing sources of funding and authorship
    • Clearly differentiating between advertisements and original content
    • Complying with universal accessibility guidelines (eg, Section 508 of the Amendment to the Rehabilitation Act of 1973)
    • Providing simple search and print functionality
    • Minimizing scientific and technical jargon

    Limitations

    This study has several limitations. First, the samples of this study were selected from Alexa Top Sites—health category. The research team had no control over the quality of the website rankings performed by Alexa.

    Additionally, some of the usability principles might change over time. For example, in this study, only 24 out of 100 websites (24.0%) followed the principle of minimizing vertical scrolling. However, with the proliferation of mobile and responsive design technology, users are becoming more accustomed to navigating websites by scrolling. Minimizing vertical scrolling may not remain a usability principle in the future.

    Finally, the survey instrument is somewhat subjective, especially for objective HC/HIT-8.2. Although the research team controlled the reliability by measuring the IRR for several websites in the sample, there was still variation across reviewers that may affect the results.

    Conclusions

    The quality and accessibility of Web-based health information is a key factor in improving access to health services and facilitating informed health decision making. The criteria proposed by ODPHP provide methods to assess the quality of health-related websites and provide baseline data and targets to drive quality improvement. In addition to having implications for website developers and policy makers, this work also points to the need for consumer education related to the quality of Web-based health information. Such education efforts are critical in a time when nearly 3 in 4 Internet users are looking for health information online [42].

    To promote increased quality of health-related websites, ODPHP updated and published Health Literacy Online: A Guide to Simplifying the User Experience, second edition [43]. Health Literacy Online is based on literature related to cognitive processing and online behavior and on usability research with more than 800 participants. It features actionable information that website owners, content writers, designers, and developers can use to create quality health websites. The recommendations in Health Literacy Online provide a clear road map for achieving the Healthy People 2020 objectives to increase the proportion of quality health-related websites (Objective HC/HIT-8). The ODPHP is working with other federal agencies to adopt the principles in Health Literacy Online and is presenting these strategies for improving health websites at national conferences.

    Acknowledgments

    This research was funded by the US Department of Health and Human Services' Office of Disease Prevention and Health Promotion (ODPHP). The ODPHP, the Centers for Disease Control and Prevention’s Office of the Associate Director for Communication, and the Office of the National Coordinator for Health Information Technology co-lead the Healthy People 2020 Health Communication and Health Information Technology topic area. The ODPHP contracted with CommunicateHealth, Inc., to conduct the data collection and analysis discussed in this paper.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Website Information Reliability Evaluation Instrument.

    PDF File (Adobe PDF File), 83KB

    Multimedia Appendix 2

    Website Usability Evaluation Instrument.

    PDF File (Adobe PDF File), 74KB

    Multimedia Appendix 3

    Sample Websites.

    PDF File (Adobe PDF File), 103KB

    References

    1. Kutner M, Greenberg E, Jin Y, Paulsen C. NCES 2006-483. Washington, DC: U.S. Department of Education, National Center for Education Statistics; 2006. The Health Literacy of America's Adults: Results from the 2003 National Assessment of Adult Literacy   URL: http://files.eric.ed.gov/fulltext/ED493284.pdf [accessed 2016-07-18] [WebCite Cache]
    2. Cline RJ, Haynes KM. Consumer health information seeking on the Internet: The state of the art. Health Educ Res 2001;16(6):671-692 [FREE Full text] [Medline]
    3. Keselman A, Browne AC, Kaufman DR. Consumer health information seeking as hypothesis testing. J Am Med Inform Assoc 2008;15(4):484-495 [FREE Full text] [CrossRef] [Medline]
    4. Diviani N, van den Putte B, Giani S, van Weert JC. Low health literacy and evaluation of online health information: a systematic review of the literature. J Med Internet Res 2015;17(5):e112 [FREE Full text] [CrossRef] [Medline]
    5. McCloud RF, Okechukwu CA, Sorensen G, Viswanath K. Beyond access: barriers to internet health information seeking among the urban poor. J Am Med Inform Assoc 2016 May 20:- Epub ahead of print. [CrossRef] [Medline]
    6. Chung M, Oden RP, Joyner BL, Sims A, Moon RY. Safe infant sleep recommendations on the Internet: let's Google it. J Pediatr 2012;161(6):1080-1084 [FREE Full text] [CrossRef] [Medline]
    7. Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA 2002;287(20):2691-2700. [Medline]
    8. Neumark Y, Flum L, Lopez-Quintero C, Shtarkshall R. Quality of online health information about oral contraceptives from Hebrew-language websites. Isr J Health Policy Res 2012;1(1):38 [FREE Full text] [CrossRef] [Medline]
    9. Fox S, Rainie L. Pew Research Center. 2002. Vital decisions: How Internet users decide what information to trust when they or their loved ones are sick   URL: http://www.pewinternet.org/files/old-media//Files/Reports/2002/PIP_Vital_Decisions_May2002.pdf.pdf [accessed 2016-04-21] [WebCite Cache]
    10. Fox S, Jones S. Pew Research Center. 2009. The social life of health information   URL: http://www.pewinternet.org/2009/06/11/the-social-life-of-health-information/ [accessed 2016-04-21] [WebCite Cache]
    11. Neter E, Brainin E. eHealth literacy: extending the digital divide to the realm of health information. J Med Internet Res 2012;14(1):e19 [FREE Full text] [CrossRef] [Medline]
    12. Tennant B, Stellefson M, Dodd V, Chaney B, Chaney D, Paige S, et al. eHealth literacy and Web 2.0 health information seeking behaviors among baby boomers and older adults. J Med Internet Res 2015;17(3):e70 [FREE Full text] [CrossRef] [Medline]
    13. Fox S, Duggan M. Pew Research Center. 2013. Health online   URL: http://www.pewinternet.org/files/old-media/Files/Reports/PIP_HealthOnline.pdf [accessed 2016-04-21] [WebCite Cache]
    14. Fiksdal AS, Kumbamu A, Jadhav AS, Cocos C, Nelsen LA, Pathak J, et al. Evaluating the process of online health information searching: a qualitative approach to exploring consumer perspectives. J Med Internet Res 2014;16(10):e224 [FREE Full text] [CrossRef] [Medline]
    15. Coulter A, Ellins J. Effectiveness of strategies for informing, educating, and involving patients. BMJ 2007;335(7609):24-27 [FREE Full text] [CrossRef] [Medline]
    16. The White House. Whitehouse.gov. 2012. Presidential Memorandum -- Building a 21st Century Digital Government   URL: https:/​/www.​whitehouse.gov/​the-press-office/​2012/​05/​23/​presidential-memorandum-building-21st-century-digital-government [accessed 2016-04-21] [WebCite Cache]
    17. WHO. 2014. The health Internet: governing principles for a .health top-level domain   URL: http://www.who.int/ehealth/programmes/governance/en/index3.html [accessed 2016-04-21] [WebCite Cache]
    18. Kitchens B, Harle CA, Li S. Quality of health-related online search results. Decision Support Systems 2014;57:454-462. [CrossRef]
    19. Kim P, Eng TR, Deering MJ, Maxfield A. Published criteria for evaluating health related web sites: review. BMJ 1999;318(7184):647-649 [FREE Full text] [Medline]
    20. Van der Vaart R, Drossaert CH, de Heus M, Taal E, van de Laar MA. Measuring actual eHealth literacy among patients with rheumatic diseases: a qualitative analysis of problems encountered using Health 1.0 and Health 2.0 applications. J Med Internet Res 2013;15(2):e27 [FREE Full text] [CrossRef] [Medline]
    21. Medical Library Association. Find Good Health Information   URL: http://www.mlanet.org/resources/userguide.html [accessed 2016-04-21] [WebCite Cache]
    22. Health on the Net Foundation. HON. 2014. The commitment to reliable health and medical information on the internet   URL: http://www.hon.ch/HONcode/Patients/Visitor/visitor.html [accessed 2016-04-21] [WebCite Cache]
    23. Consumers Union. Mission   URL: http://consumersunion.org/about/mission/ [accessed 2016-04-21] [WebCite Cache]
    24. The Agency for Healthcare Research and Quality. AHRQ. 2006. Guidelines for Ensuring the Quality of Information Disseminated to the Public   URL: http://www.ahrq.gov/research/publications/ahrq-info-quality-gdlns.html [accessed 2016-04-21] [WebCite Cache]
    25. URAC. Tspwebs. 2001. Health Web Site Standards, Version 1.0   URL: http://webs.tspwebs.net/URAC%20Standards.pdf [accessed 2016-04-27] [WebCite Cache]
    26. Health Improvement Institute. 2005. Health Website Ratings Instrument, Version 2.0   URL: https:/​/dl.​dropboxusercontent.com/​content_link/​XcTWVhlqdUN8rYGqTRDc4uLTG2NOHy54swzmXbVvrgzm619tqqYNeQSgfsgMMFGi/​file [accessed 2016-04-27] [WebCite Cache]
    27. The U.S. Department of Health Human Services. Research-Based Web Design & Usability Guidelines (Enlarge/E). Washington, DC: U.S. Government Printing Office; 2006.
    28. Travis D. Userfocus. 2014. 247 web usability guidelines   URL: http://www.userfocus.co.uk/resources/guidelines.html [accessed 2016-04-21] [WebCite Cache]
    29. UsabilityNet. 2006. Design guidelines for the Web   URL: http://www.usabilitynet.org/tools/webdesign.htm [accessed 2016-04-21] [WebCite Cache]
    30. Sharpened Productions. 2015. 10 Rules of Web Design   URL: http://sharpened.com/web_design_rules [accessed 2016-04-21] [WebCite Cache]
    31. Krug S. Don’t Make Me Think, Revisited: A Common Sense Approach to Web Usability. 3rd edition. Berkeley, CA: New Riders; 2014.
    32. Redish J. Letting Go of the Words: Writing Web Content that Works (Interactive Technologies). 2nd edition. Burlington, MA: Morgan Kaufmann; 2012.
    33. Lee K, Hoti K, Hughes JD, Emmerton LM. Consumer Use of “Dr Google”: A Survey on Health Information-Seeking Behaviors and Navigational Needs. J Med Internet Res 2015;17(12):e288 [FREE Full text] [CrossRef] [Medline]
    34. Weymann N, Härter M, Dirmaier J. Quality of online information on type 2 diabetes: a cross-sectional study. Health Promot Int 2015;30(4):821-831. [CrossRef] [Medline]
    35. Stanford J, Tauber ER, Fogg BJ, Marable L. Ebusiness Forum. 2002. Experts vs. Online Consumers: A Comparative Credibility Study of Health and Finance Web Sites   URL: http://www.ebusinessforum.gr/old/content/downloads/comparativeCredibilityStudy.pdf [accessed 2016-04-15] [WebCite Cache]
    36. Sillence E, Briggs P, Harris PR, Fishwick L. How do patients evaluate and make use of online health information? Soc Sci Med 2007;64(9):1853-1862. [CrossRef] [Medline]
    37. Agree EM, King AC, Castro CM, Wiley A, Borzekowski DL. “It’s Got to Be on This Page”: Age and Cognitive Style in a Study of Online Health Information Seeking. J Med Internet Res 2015;17(3):e79 [FREE Full text] [CrossRef] [Medline]
    38. Alexa. 2014. Top Sites -- health category   URL: http://www.alexa.com/topsites/category/Top/Health [accessed 2014-10-15] [WebCite Cache]
    39. Alexa. 2016. How are Alexa’s traffic rankings determined?   URL: https://support.alexa.com/hc/en-us/articles/200449744-How-are-Alexa-s-traffic-rankings-determined- [accessed 2016-06-10] [WebCite Cache]
    40. Rippen H, Risk A. e-Health Code of Ethics (May 24). J Med Internet Res 2000;2(2):E9 [FREE Full text] [CrossRef] [Medline]
    41. Altman D. Practical Statistics for Medical Research. Boca Raton, FL: CRC Press; 1990.
    42. Pew Research Center. 2013. Health Fact Sheet   URL: http://www.pewinternet.org/fact-sheets/health-fact-sheet/ [accessed 2016-04-21] [WebCite Cache]
    43. U.S. Department of Health and Human Services, Office of Disease Prevention and Health Promotion. Health.gov. 2015. Health Literacy Online: 2nd Edition   URL: http://health.gov/healthliteracyonline/ [accessed 2016-06-10] [WebCite Cache]


    Abbreviations

    HC/HIT: Health Communication and Health Information Technology
    ICC: interclass correlation
    IRR: interrater reliability
    ODPHP: Office of Disease Prevention and Health Promotion


    Edited by G Eysenbach; submitted 19.05.16; peer-reviewed by K Summers, G Reddish; comments to author 08.06.16; revised version received 22.06.16; accepted 23.06.16; published 02.08.16

    ©Theresa Devine, Jordan Broderick, Linda M Harris, Huijuan Wu, Sandra Williams Hilfiker. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 02.08.2016.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.