Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, March 11, 2019 at 4:00 PM to 4:30 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 02.05.19 in Vol 21, No 5 (2019): May

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/12522, first published Oct 19, 2018.

This paper is in the following e-collection/theme issue:

    Review

    Consumer Evaluation of the Quality of Online Health Information: Systematic Literature Review of Relevant Criteria and Indicators

    School of Information, The University of Texas at Austin, Austin, TX, United States

    Corresponding Author:

    Yan Zhang, PhD

    School of Information

    The University of Texas at Austin

    1616 Guadalupe Street Suite #5.202

    Austin, TX, 78701

    United States

    Phone: 1 5124719448

    Email: yanz@ischool.utexas.edu


    ABSTRACT

    Background: As the quality of online health information remains questionable, there is a pressing need to understand how consumers evaluate this information. Past reviews identified content-, source-, and individual-related factors that influence consumer judgment in this area. However, systematic knowledge concerning the evaluation process, that is, why and how these factors influence the evaluation behavior, is lacking.

    Objective: This review aims (1) to identify criteria (rules that reflect notions of value and worth) that consumers use to evaluate the quality of online health information and the indicators (properties of information objects to which criteria are applied to form judgments) they use to support the evaluation in order to achieve a better understanding of the process of information quality evaluation and (2) to explicate the relationship between indicators and criteria to provide clear guidelines for designers of consumer health information systems.

    Methods: A systematic literature search was performed in seven digital reference databases including Medicine, Psychology, Communication, and Library and Information Science to identify empirical studies that report how consumers directly and explicitly describe their evaluation of online health information quality. Thirty-seven articles met the inclusion criteria. A qualitative content analysis was performed to identify quality evaluation criteria, indicators, and their relationships.

    Results: We identified 25 criteria and 165 indicators. The most widely reported criteria used by consumers were trustworthiness, expertise, and objectivity. The indicators were related to source, content, and design. Among them, 114 were positive indicators (entailing positive quality judgments), 35 were negative indicators (entailing negative judgments), and 16 indicators had both positive and negative quality influence, depending on contextual factors (eg, source and individual differences) and criteria applied. The most widely reported indicators were site owners/sponsors; consensus among multiple sources; characteristics of writing and language; advertisements; content authorship; and interface design.

    Conclusions: Consumer evaluation of online health information is a complex cost-benefit analysis process that involves the use of a wide range of criteria and a much wider range of quality indicators. There are commonalities in the use of criteria across user groups and source types, but the differences are hard to ignore. Evidently, consumers’ health information evaluation can be characterized as highly subjective and contextualized, and sometimes, misinformed. These findings invite more research into how different user groups evaluate different types of online sources and a personalized approach to educate users about evaluating online health information quality.

    J Med Internet Res 2019;21(5):e12522

    doi:10.2196/12522

    KEYWORDS



    Introduction

    More than 70% of US adults search online for health information [1]. The information found online shapes and influences consumers’ health beliefs, intentions, health behaviors, and health care decision making [2-5]. Since the inception of the internet, the quality of health information has been a source of concern for stakeholders due to the unregulated nature of the medium [6]. This concern is furthered by the fast growth of social media and user-generated content and corroborated by more than 200 evaluation studies conducted by subject experts, which collectively suggest that the quality of consumer-oriented health information on the internet varies greatly and that the overall quality was low [7] and remains low [8].

    Making decisions based on low-quality health information (eg, information that is inaccurate, incomplete, or biased) may lead to harmful consequences, such as delayed treatment or extreme anxiety [9], and subsequently increase consumer vulnerability [10,11]. Nevertheless, evaluating the quality of information has been a major challenge for online health consumers [12-14]. For example, some consumers are uncertain about the accuracy, completeness, and validity of the information they encounter [15,16]; some cannot differentiate between scientific facts, empirical factors, and personal opinions [17]; and others suffer from information overload and subsequently lack the confidence and ability to evaluate information [18-21]. Studies have found that compared to health care providers or information professionals, consumers tend to give higher quality ratings to health information from both traditional health websites [22] and social media sites [23].

    The ability to critically evaluate the quality of health information is an important component of health literacy [10], which is an important determinant of health [24]. To enhance this ability (and related skills), it is necessary to understand how consumers evaluate the quality of health information in the current internet environment. Consumer evaluation is subjective, driven by one’s information needs. Therefore, as a starting point, we adopted a broad conceptualization that defines quality through “fitness for use” [25]: Information is of good quality when it serves users’ needs. It is worth noting that this concept of quality is described using different terms in the existing literature, including but not limited to quality, credibility, trust, reliability, believability, and usefulness. In this review, we included articles using all these terms. We chose to be inclusive, because we want to achieve a comprehensive view of the assessments that consumers perform in the process of determining whether they would be willing to use a piece of information.

    Guided by this understanding of quality, three recent systematic reviews were identified as relevant to our current research: One review focused on identifying factors that impact consumer judgment of trustworthiness and credibility of online health information [11], the second one identified the antecedents of trust in health information websites [27], and the final one reviewed the association between low health literacy and perceived quality and trust in online health information and low literacy consumers’ ability to evaluate information quality [10]. These reviews revealed that consumers’ quality evaluation is influenced by both source- and content-related factors [10,11,27]. Examples of source-related factors are website design (eg, layout, visual design, and interactive features), loading speed, and the authority of the owner or sponsor [11,27-30]. Examples of content-related factors are the authority of the author, content readability, content organization, use of evidence and citations, and the appearance of advertisements [11,27,31-33]. Additionally, a number of individual characteristics were identified as influencers, including demographics (eg, age, gender, and educational attainment), perceived health status, knowledge about the content, health beliefs, and level of health literacy [10,11,22,27,31].

    These reviews provide an informative overview of factors that influence consumer online health information evaluation behavior but shed limited light on why and how these factors influence the evaluation behavior. From the perspective of information seeking, evaluation of information is a judgment and decision-making process that precedes users’ acceptance or rejection of received information [34]. Judgment and decision making involve applying certain criteria, principles, or standards to form evaluations [35]. Thus, to understand consumer quality evaluation behavior, it is necessary to understand the criteria used to guide the evaluation. Among the previously mentioned systematic reviews, only one [10] summarized the evaluation criteria reported in five studies on consumers with low health literacy. A more comprehensive understanding of the evaluation criteria is needed. This review intends to fill this gap.

    Evaluation of the quality of online health information is a process of applying criteria to evaluate information. Thus, in addition to applying criteria, we need a better understanding of how consumers perceive online information. To achieve this goal, we deliberately differentiate between two concepts: criteria and indicators. Criteria are rules or filters that people apply to an information object to assess its value or worth [36]. Indicators, also termed cues or markers [37], are perceivable elements associated with an information object that allow people to reflect on the quality of the object [8]. Criteria are abstract, reflecting one’s values and preferences and mediating information selection decisions. Indicators are affordances of information objects that trigger or support the application of the criteria. Criteria are comparatively stable, whereas indicators are amenable to change. New indicators could emerge, and old ones could disappear with the development of new technologies and design preferences.

    In this article, we focus on the following research questions: (1) What criteria do consumers use to evaluate the quality of online health information? (2) What elements of information objects do consumers use as quality indicators? (3) Which indicators convey positive evaluations and which convey negative evaluations? (4) What is the relationship between indicators and criteria, that is, what criteria do each indicator correspond to? We argue that a more comprehensive understanding of criteria used in the evaluation process can bring some clarity to the dimensions of quality perceived by online health consumers as well as their quality evaluation process. By explicating the relationship between indicators and criteria and identifying positive and negative judgments that indicators convey, the results can also inform the design of more user-friendly health information content and information systems.


    Methods

    Search Strategies

    Seven online databases, including PubMed, Web of Science, PsycINFO, CINAHL (Cumulative Index to Nursing and Allied Health), Cochrane Library, Library and Information Science Source, and Communication and Mass Media Complete, were searched in July 2017 to obtain relevant journal articles. These databases were chosen because they cover major academic disciplines that study consumer online health information search, including health, information and library science, psychology, and mass communication. Keywords, including quality, credibility, trust, reliability, accuracy, readability, relevance, and usefulness were used in combination with the keywords consumer or patient and online health information evaluation or online health information assessment. After the searches, we manually screened the references to identify relevant articles and further examined the reference lists of these articles. Additionally, we examined the references cited in the three systematic reviews mentioned above and articles that cited these reviews (using Google Scholar’s “cited by” function).

    Inclusion/Exclusion Criteria

    Articles meeting the following criteria were eligible for inclusion in this review: (1) The study primarily focused on consumer evaluation of health information on the internet. Health consumers include patients, caregivers, and the general public who sought or were interested in seeking health information. This focus differentiates this review from prior reviews of health care professionals or expert evaluation of online health information for consumers [7,8]. Articles that focus on media other than the internet (eg, TV and radio) were excluded. (2) The study was empirical and based on direct inquiries with health consumers where criteria were described by participants and not imposed by researchers. Articles that used only predefined evaluation criteria to survey consumers or analyze their responses without allowing new criteria to emerge were excluded. We also excluded correctional studies that focus on identifying factors (eg, source expertise) influencing consumer evaluation behavior but do not provide additional results on how quality evaluation is performed. (3) The article was published after 2002, when research on consumer evaluation of online health information began to emerge. (4) The article was written in the English language. (5) The article was published in a peer-reviewed journal.

    Study Identification

    Figure 1 shows the process involved in identifying eligible studies. Three authors (YS, YZ, and JG) reviewed a subset of the search results by reading titles and abstracts. YS and YZ both reviewed 10% of the records (256 records in total) to check the intercoder agreement in filtering potentially relevant articles (Cohen kappa=0.83). Both YS and YZ screened the full-text articles. When there was uncertainty involved in excluding a full-text article, the other two authors provided their input.

    Data Extraction and Analysis

    Full text of the 37 selected articles was imported into MAXQDA 12 (VERBI Software GmbH, Berlin, Germany) for analysis. We extracted the following information: basic characteristics of the articles (eg, year of publication, country of origin, health topics, and aims of the study), research methods, sampling techniques, participant characteristics (eg, demographics and disease experiences), source studied (eg, the internet or specific health websites), and characteristics of the search tasks (eg, self-generated vs assigned) when search tasks were involved. Guided by their corresponding definitions, indicators and the corresponding criteria were extracted from the results and discussion reported in the original papers. When no clear relationships were reported (in most of such cases, indicators were reported without mentioning the criteria. For example, .com was reported as a negative indicator of quality, but criteria by which this judgment was reached were not reported), the authors of the review derived the relationships from the participants’ direct quotes reported in the original papers, the original authors’ discussion of the results, or the interpretation of the authors of the review. Indicators were further coded into positive (+, entailing positive quality judgment), negative (–, entailing negative judgment), or both (±, entailing both positive and negative judgments). When participants commented on the absence of an indicator (eg, no author credential or no advertisements), it was coded as positive if the absence implies low quality and as negative if the absence implies high quality. The criteria were also coded into the three categories based on their correspondence with indicators.

    We analyzed the basic characteristics of the included studies using descriptive statistics. The qualitative content analysis method [26] was used to identify themes and build categories based on the extracted information concerning criteria and indicators in an iterative manner. YS coded all the articles. YZ validated the results by comparing each assigned code to the full-text of the articles. A number of group meetings were held to discuss the codes, especially relationships between indicators and criteria. Discrepancies were discussed among all authors.

    Figure 1. Article screening process.
    View this figure

    Results

    Basic Characteristics of the Included Articles

    The 37 articles included in the review were published between 2002 and 2017. They originated from 8 countries, primarily United Kingdom (N=12), United States (N=11), Australia (N=4), and the Netherlands (N=3). The characteristics of each included article are summarized in Table 1.

    Focus groups (n=17), interviews (n=16), and observations of participants performing predefined (n=11) or self-generated (n=6) search tasks were the primary research methods used in the selected articles. Observations were often used with other methods including think aloud, guided interviews, focus groups, or diaries. Fourteen articles used multiple research methods.

    Twenty-one articles focused on information on a specific health condition or issue (eg, HIV prevention, diabetes, disabilities, and chronic diseases), and the remaining articles did not specify a subject focus. Twelve studies recruited patients with a specific condition, and the others recruited people who had searched online for health information (n=6) or had a strong interest in their health or a particular condition (n=5). Twenty-eight articles involved adult participants (≥18 years old), of which 10 articles also involved older adults (>64 years old). Four studies included adolescents aged 11-17 years. The number of participants ranged from 5 to 188 (median=21). In terms of sampling technique, 26 articles used purposive sampling, five used convenience sampling, and the remaining six did not report the sampling methods.

    Table 1. Characteristics of the included articles.
    View this table

    Regarding evaluation of internet sources, 28 articles did not specify a scope. The remaining nine articles specified or preselected sources for evaluation (eg, pediatric sun protection websites, the National Institutes of Health website, the Centers for Disease Control and Prevention website, patients.co.uk, and online forums).

    Quality Evaluation Criteria Used by Consumers

    Twenty-five criteria were identified (Table 2). The definitions were derived from the codes or drawn directly from the included studies.

    Among these criteria, trustworthiness, expertise, and objectivity were reported most often in the articles, followed by transparency, popularity, and understandability. Eight criteria including relevance, familiarity, accessibility, identification, believability, accuracy, readability, and currency were reported in 10-15 articles. The remaining 11 criteria appeared in less than 10 articles.

    Quality Indicators Used by Consumers

    Indicators used by consumers to evaluate the quality of online health information were related to three aspects of online information: source, content, and design. Table 3 shows their distribution across the three categories.

    About 52% of the indicators were content related, followed by design (25%) and source factors (23%); 69% of the indicators were associated with positive quality judgment, 21% were associated with negative quality judgment, and 10% could lead to both positive and negative judgment.

    Table 2. Criteria used by consumers to evaluate the quality of online health information.
    View this table
    Table 3. Distribution of quality indicators used by consumers to evaluate the quality of online health.
    View this table

    Source

    Source is the entity that creates, hosts, or distributes content. A source can be a website or the owner, creator, or sponsor of the site. Six categories of source-related quality indicators were identified: site owners/sponsors, site types, disclosures, third-party accreditations, recommendations from other systems or users, and website scope. More detailed indicators reported in the included articles, their direction of influence on quality judgment (positive, negative, or both), the corresponding criteria that guide the consumers’ appraisal of the indicators, as well as the value of the criteria (positive, negative, or both) are shown in Table 4. The indicators in the tables are self-explanatory; therefore, we focus on describing the most frequently appearing indicators in the included studies and indicators that can lead to both positive and negative judgments.

    The Most Frequently Mentioned Indicators

    The most frequently mentioned source-related indictors were site owners/sponsors, with sites run by reputable organizations, educational and academic institutions [18,40,41,46, 52,59], and medical experts and health institutions [32,39,44,46,51, 54,55,57,59,65] being considered more trustworthy and offering higher levels of expertise. The second most frequently reported indicators were about disclosure. Sites that disclose their motivations were highly valued [40,42,48,64,66,67], whereas a lack of a clear statement of purpose and motivation damaged trust [49]. The third mostly frequently reported indicators were recommendations from other systems or users. High ranks in search engines [13,52,63] and a large number of visitors or followers [61,63] were viewed as indicators of high site popularity, and subsequently, high quality. In addition, sites linked from or recommended by a trusted website [30,43,53] or trusted others (eg, health care providers, families, and friends) [20,41,47,61,67] were considered trustworthy.

    Indicators With Both Positive and Negative Influences on Evaluation

    Mixed attitudes were found toward some indicators representing site owners/sponsors. First, most participants believed that government websites (eg, National Health Service and Centers for Disease Control and Prevention) reflect high levels of expertise and good intentions [39-41,46,53,54,58,59,64,65]; however, some consumers suspected that the information on government websites is biased due to their agendas [41,52], and some, particularly younger generations, did not identify themselves with government sources, considering them “less cool” and not relatable [58]. Second, most people considered sites operated by local health societies to have a high level of expertise; however, some minorities and people from nonmainstream cultures (eg, aboriginal communities) were likely to question the relevance and accuracy of the information from these sites [47]. Third, people usually considered websites owned by commercial companies less objective [42,46,48] and trusted more websites with no commercial interests [18,20,42,55,67]; nevertheless, popular commercial websites such as BabyCenter.com, ParentsPlace.com, and WebMD.com were favored by some people for their expertise and comprehensiveness [42]. Fourth, a few people viewed information from pharmaceutical company websites as “official” [40], whereas others considered their information biased due to the financial interests involved [32,40,46,48].

    Consumers had mixed attitudes toward the website types, particularly social media sites. Some consumers favored online discussion groups, chat rooms, and listservs because they offered first-person narratives and practical information and support from peers with whom they could identify (ie, those who have similar conditions) [46,53,57], but some disliked such sites for their lack of objectivity and expertise [13,53,59]. Concerning Wikipedia, some people questioned its objectivity because information can be edited by anyone on the Web [50,58,61], but some consumers were attracted to its encyclopedic nature and comprehensiveness [63].

    Consumers also had different opinions regarding sites recommended by others. Some trusted a site recommended by trusted others (eg, health care providers, families, and friends) [20,41,47,61,67]; however, some consumers recognized that recommendations from other individuals may not be relevant to their situation [67].

    Table 4. Evaluation of the source.
    View this table

    Content

    Content refers to the information contained in a source as well as the presentation of the information. Eight categories of content-related indicators were identified: substance, writing and language, presentation, references, authorship, audience, date/updating, and advertisements. Table 5 shows these indicators, the corresponding criteria that guide the consumers’ appraisal of the indicators, and their influence on quality judgment.

    The Most Frequently Mentioned Indicators

    The most frequently reported content indicators were about consensus among sources. Content that appears in multiple sources, be it online sources, sources in other media (eg, newspaper, television, books, and academic journals), or health care professionals, is trusted by consumers. Writing- and language-related factors were the second most frequently reported content indicators. Consumers expect high-quality information to be error free in spelling and grammar, use straightforward language, and have a clear layout. The third most frequently reported indicators were advertisements. Consumers expect quality websites to neither depend on advertisements [63] nor seek to make a profit [62]. Therefore, sites with advertisements were considered less objective [46,49,54,63,64], be it in the form of commercial links [46], advertisement banners [30,32,55], popups [32,40], or other formats.

    Indicators With Both Positive and Negative Influences on Evaluation

    Consumers had mixed attitudes toward two content types: objective facts and personal experiences. Some consumers value objective facts [43,45,62], whereas some dissatisfy with information that contained solely objective facts, considering it unbalanced [69]. With regard to personal experiences, some consumers favored first-hand experiences, stories, and advice (eg, recommendations for medical gadgets, meal planning and exercising, and advice on completing medical benefit forms) from people with similar conditions for their practicality [40,41,45,46,49,63,64], but some had concerns that personal experiences lack objectivity, especially when it merely is a personal opinion [30,69].

    Celebrity endorsement was also an indicator with both positive and negative influences on quality evaluation. Some trusted the endorsed information but others question its objectivity due to the potential financial interest involved [61,67].

    The use of medical and technical vocabularies raised contention among consumers as well. For some consumers, high-quality information was easy to understand, that is, it exhibited less use of professional medical vocabularies [30,43,49,64,69] or provided easy-to-understand definitions of medical jargon [46,49], especially for educational and government sites [18,42,52]; however, for others, the use of technical vocabularies demonstrated expertise and was highly valued [51,60,63].

    Some consumers doubted information (especially diagnosis and treatment information) authored by other unknown consumers [42,53], but others tended to trust content written by their peers because of similar demographic or health characteristics [32].

    For health interventions, some consumers examined the release time and perceived newer interventions to have higher quality than the existing ones because the new intervention would have undergone more testing and research, whereas others were uncertain about the reliability of newer interventions [67].

    Design

    Design refers to the appearance of a website or an app and the interactions that it affords. Four categories of design-related quality indicators were identified: interface design, interaction design, navigation design, and security settings. Table 6 shows the specific indicators, the corresponding criteria that guide the consumers’ appraisal of the indicators, and their influence on quality judgment.

    Table 5. Evaluation of content.
    View this table
    Table 6. Evaluation of design.
    View this table
    The Most Frequently Mentioned Indicators

    The most frequently reported design indicators were related to interface design, mostly visual factors, including the overall appearance of a site, the graphics it includes, and font size. Interaction design features, including links, interactive functions, and other interactive features (eg, loading time and login requirement), were the second most frequently mentioned quality indicators. Sites with robust search capabilities (eg, easy to locate and diverse search entrance), offering useful tools (eg, self-management tools), and rendering smooth user-system interaction (eg, providing links to additional relevant sources and not having pop-ups) were perceived as high quality. Navigation-related indicators such as navigation aids and site maps were the third most frequently mentioned quality indicators.

    Indicators With Both Positive and Negative Influences on Evaluation

    Mixed opinions existed concerning the interactive functions of self-management and assessment tools (eg, health calculators). Some consumers valued tailored results and advice [46,49], but some questioned the accuracy and objectivity of the information generated [46,62].

    Individual Factors Influencing Quality Judgment

    In addition to source-, content-, and design-related factors, the evaluation of online health information quality was also affected by individual factors including individuals’ personal situation, prior knowledge or experience of a source, personal knowledge and beliefs, and intuition. Table 7 shows the specific factors, the corresponding criteria that guide the consumers’ appraisal, and their influence on quality judgment.

    The Most Frequently Mentioned Factors

    Individuals’ prior knowledge and experience of a source were mentioned most frequently as factors that influence quality judgment. Consumers tended to trust sites that they had experience with [49,63], because they may already know the source to be credible [13,18,39,50,53,59,65], have had positive experiences with it [42,54,67], have seen it from advertisements on other media (eg, television and magazine) [42,58], or are familiar with the organization behind the source [18,42].

    The category of personal situation was the second most frequent factor. Information relevant to individuals’ search topics (eg, hormone replacement therapy) [32,45], needs and goals (eg, offering easy reading level message for younger people) [40,54,57], specific circumstances (eg, localization) [40,45,62,64], and experiences and symptoms [18,56,62,64] was considered to be of high quality.

    The other two categories of individual factors were mentioned with the same frequency. One category is personal knowledge and beliefs. Consumers highly valued information consistent with their own beliefs and knowledge [18,32,41,56,63,64]. The other category is intuition. Some consumers undertook “subconscious filtering” to filter out potential political and gender biased information [69], and some consumers relied on common sense [39], sensation [63], instinct, or “gut feelings” [55,65,66] to evaluate information.

    Table 7. Individual factors.
    View this table

    Discussion

    In this article, we reviewed 37 empirical studies that reported consumers’ accounts of how they evaluate the quality of online health information. This review extends the existing literature by making two major conceptual contributions. First, it offers a clear conceptual understanding of the dimensions of quality of online health information perceived by consumers by differentiating criteria from indicators. Second, it explicates the relationship between webpage quality indicators (webpage elements) and the quality judgment by differentiating positive and negative influences that indicators have on judgment. In this section, we discuss each contribution and then outline practical implications and limitations of this review.

    Dimensions of Online Health Information Quality

    In the existing literature, quality was often defined and assessed differently. We guided the article selection for the review using a general conceptualization that defines quality as “fitness for use” [25]. Other authors have offered more specific conceptualizations. For example, Rieh [70] assessed quality as the extent to which users think that the information is useful, good, current, and accurate. Bates et al [71] measured health information quality in terms of its trustworthiness, truthfulness, readability, and completeness. Benotsch et al [22] rated the quality of health websites on five dimensions: accuracy, amount of detail, trustworthiness-credibility, relevance, and usefulness. Eastin [72] rated the credibility of health information on three dimensions: accuracy, believability, and factualness. The lack of consistency in measuring online health information quality suggests that there is a lack of clear conceptual understanding of what information quality means to online health consumers.

    By clearly differentiating quality judgment criteria (rules that reflect notions of value and worth) and indicators (properties of information objects to which criteria are applied to form judgments) reported in the included studies, this review identified 25 dimensions (criteria) along which consumers evaluate the quality of online health information (Table 2). Because the included articles differ on aspects such as health issues of concern, participant demographics, and sources examined, this wide range of criteria reported and the uneven distribution of the criteria across the included articles suggest that consumer evaluation of online health information may be influenced by contextual factors such as user characteristics, health conditions, and online sources. In addition to these factors, the current review, consistent with prior reviews [10,11,27], also identified a range of individual factors that influence quality judgment behavior, such as prior experience with a source and personal knowledge and beliefs. Therefore, future studies should attempt to identify the most influential contextual factors (including individual factors) that affect consumers’ application of quality criteria to further enhance the theoretical understanding of this behavior. Empirical studies of consumer online health information evaluation should also consider these contextual factors in research design.

    Despite the wide range, however, three criteria (trustworthiness, expertise, and objectivity) were reported in 31 articles, indicating that they are used consistently across user groups, source types, and health conditions and that they constitute core dimensions of online health information quality as perceived by consumers. The fact that trustworthiness and expertise are primary dimensions is consistent with general media source credibility research [73]. It is not surprising that objectivity, that is, whether a source or information presents objective factual or evidence-based information, is also important for health information. Three additional criteria—transparency (reported in 21 articles), popularity (reported in 19 articles), and understandability (reported in 18 articles)—are also commonly reported and could be viewed as secondary dimensions of online health information quality. These findings imply that consumers’ perceived online health information quality could be reasonably measured by a small set of core dimensions.

    Relationship Between Quality Indicators and Quality Judgment

    Previous reviews summarized indicators used by consumers to evaluate the quality of online information [10,37]. Sbaffi and Rowley [11] further reported the direction of the effect (ie, positive vs negative) of the (design and content) indicators. However, the situational nature of the relationship between indicators and quality judgment, that is, the fact that their relationship is not one-on-one, but dependent on users’ values and the criteria applied, was not explicitly discussed. For example, government institutions, usually associated with high level of expertise and authority, are perceived by some consumers as biased sources with which they have a hard time relating. The other example is that consumer-generated content (eg, personal blogs and listserves) indicates low objectivity and low level of expertise to some consumers, but to others, it is considered highly practical and relatable. Thus, a unique contribution of this review is that it clearly maps out the direction of the impact (ie, positive or negative) of a number of indicators on quality judgment and the underlying reasons (ie, criteria) for the impact.

    Practical Implications

    The identification and differentiation of positive and negative indicators provide clear guidance for online health information designers. They can incorporate positive indicators (eg, offering authors’ credentials and presenting information in a clear and organized way) and avoid negative indicators (eg, dead links and flash media format) to offer users better information seeking experiences. The fact that the same indicator (eg, government institutions as the source owner) can lead to different quality judgment for different people suggests that designers should also carefully investigate target users’ values and the corresponding criteria that they use to evaluate health information. This calls for active user research and user involvement in the design process.

    The results of the review also have implications for consumer education. The review revealed a wide range of criteria that consumers use to evaluate the quality of online health information. Many of the criteria, such as familiarity, identification, relevance, practicality, and usefulness, are highly subjective and situational, influenced by factors such as information needs, online information search experience, and personal beliefs. In some cases, consumers assign such criteria higher priority than more objective ones such as expertise [57]. The review also revealed that consumers use a diverse set of quality indicators. The implications of some of the indicators are not well understood. For example, some consumers believe that the appearance of copyright information or the word “clinical” indicates high information quality [61,67]. Some consumers view the fact that a website passes the screening of virus/security software as an indicator of high quality [57]. There are also consumers assuming that third-party accreditations are indicators of information accuracy, when, in fact, the guidelines that these accreditations follow do not really check for information accuracy [74,75]. Consumers need education to use more objective criteria to evaluate online health information and understand the implications of a number of quality indicators.

    Limitations

    This review has several limitations. First, we selected only studies where consumers explicitly described their quality evaluation behavior. These studies tend not to ask consumers to rate criteria or indicators; thus, we could not identify the importance of each indicator or criterion in quality judgment. Future reviews are needed to fill this gap. Second, we did not differentiate and compare results based on observations and results drawn from verbal inquiries as few included studies did. Eysenbach and Kohler [30] reported discrepancies between participants’ verbal accounts of what they do to evaluate health information and what they actually did in performing search tasks (based on observations). Thus, future empirical studies are needed to shed light on this gap. Third, in the coding process, we used criterion and indicator terms from the original papers, where feasible. In cases where we needed to infer criteria from indicators, we followed the mostly commonly recognized categorization by referring to prior empirical research and reviews or inferred the criteria from participants’ quotes. However, due to the different perspectives of the authors of the original papers and the inherent overlap between terms, such as comprehensiveness and completeness, our syntheses are inevitably affected by a certain degree of subjectivity. Fourth, because most studies treated the internet as one source of information without differentiating source types (eg, regular websites and social media), we were not able to identify whether the use of evaluation criteria and indicators differs by source type.

    Conclusions

    The quality of online health information is a complex concept involving more than two dozen dimensions, as perceived by consumers. Although a set of core dimensions can be identified, the diversity involved in consumers’ use of criteria is too obvious to ignore. Further examination of contextual factors (eg, different source and user characteristics) that influence consumers’ application of quality criteria will bring further clarity to the concept. The review identified 165 indicators, to which criteria are applied to reach a quality judgment. Indicators could be source, content, or design related; they can have a positive or negative impact on quality judgment, contingent on situations and users’ values and beliefs. The identification and differentiation of positive and negative indicators along with their respective criteria can provide clearer guidance for designers of online health websites and educational interventions. Compared to experts’ evaluation, consumers’ evaluation of online health information relies heavily on peripheral cues and is influenced by various contextual factors (eg, personal beliefs and information needs). This finding suggests that current quality evaluation checklists, which are mostly based on experts’ view of quality, may not effectively serve the needs of consumers. Consumer behavior needs to be considered in the design of interventions that intend to promote quality evaluation in online searches. At the same time, it is worth noting that criteria and indicators used by consumers merit critical evaluation, as some criteria are overly subjective and the implications of some indicators are not well understood. User education is needed to address user misconceptions and the associated suboptimal evaluation behavior.

    Acknowledgments

    The project is supported by funding from the Portuguese Foundation for Science and Technology and the Digital Media Program at UT-Austin and the UT School of Information Governor Bill Daniel Fellowship.

    Conflicts of Interest

    None declared.

    References

    1. Fox S, Duggan M. Pew Research Center Internet & Technology. 2013. Health Online 2013   URL: https://www.pewinternet.org/2013/01/15/health-online-2013/ [accessed 2019-04-18] [WebCite Cache]
    2. Kealey E, Berkman CS. The relationship between health information sources and mental models of cancer: findings from the 2005 Health Information National Trends Survey. J Health Commun 2010;15 Suppl 3:236-251. [CrossRef] [Medline]
    3. Fox S. Pew Research Center. 2011. The Social Life of Health Information 2011   URL: http://www.pewresearch.org/fact-tank/2014/01/15/the-social-life-of-health-information/ [accessed 2018-10-16] [WebCite Cache]
    4. Hesse BW, Nelson DE, Kreps GL, Croyle RT, Arora NK, Rimer BK, et al. Trust and sources of health information: the impact of the Internet and its implications for health care providers: findings from the first Health Information National Trends Survey. Arch Intern Med 2005 Dec;165(22):2618-2624. [CrossRef] [Medline]
    5. McKinley CJ, Wright PJ. Informational social support and online health information seeking: Examining the association between factors contributing to healthy eating behavior. Computers in Human Behavior 2014 Aug;37:107-116. [CrossRef]
    6. Eng TR, Maxfield A, Patrick K, Deering MJ, Ratzan SC, Gustafson DH. Access to Health Information and Support:A Public Highway or A Private Road? JAMA 1998 Oct 21;280(15):1371-1375. [CrossRef]
    7. Eysenbach G, Powell J, Kuss O, Sa E. Empirical Studies Assessing the Quality of Health Information for Consumers on the World Wide Web. JAMA 2002 May 22;287(20):2691-2700. [CrossRef]
    8. Zhang Y, Sun Y, Xie B. Quality of health information for consumers on the web: A systematic review of indicators, criteria, tools, and evaluation results. J Assn Inf Sci Tec 2015 Apr 29;66(10):2071-2084. [CrossRef]
    9. Cline R, Haynes K. Consumer health information seeking on the Internet: the state of the art. Health Educ Res 2001 Dec;16:671-692. [CrossRef]
    10. Diviani N, van den Putte B, Giani S, van Weert JCM. Low health literacy and evaluation of online health information: a systematic review of the literature. J Med Internet Res 2015 May 07;17(5):e112 [FREE Full text] [CrossRef] [Medline]
    11. Sbaffi L, Rowley J. Trust and Credibility in Web-Based Health Information: A Review and Agenda for Future Research. J Med Internet Res 2017 Jun 19;19(6):e218 [FREE Full text] [CrossRef] [Medline]
    12. Arora NK, Hesse BW, Rimer BK, Viswanath K, Clayman ML, Croyle RT. Frustrated and confused: the American public rates its cancer-related information-seeking experiences. J Gen Intern Med 2008 Mar;23(3):223-228 [FREE Full text] [CrossRef] [Medline]
    13. Feufel MA, Stahl SF. What do web-use skill differences imply for online health information searches? J Med Internet Res 2012 Jun;14(3):e87 [FREE Full text] [CrossRef] [Medline]
    14. Car J, Lang B, Colledge A, Ung C, Majeed A. Interventions for enhancing consumers' online health literacy. Cochrane Database Syst Rev 2011(6):CD007092. [CrossRef] [Medline]
    15. Sweet SN, Perrier M, Podzyhun C, Latimer-Cheung AE. Identifying physical activity information needs and preferred methods of delivery of people with multiple sclerosis. Disabil Rehabil 2013;35(24):2056-2063. [CrossRef] [Medline]
    16. Unruh HK, Bowen DJ, Meischke H, Bush N, Wooldridge JA. Women's approaches to the use of new technology for cancer risk information. Women Health 2004;40(1):59-78. [CrossRef] [Medline]
    17. Gray NJ, Klein JD, Cantrill JA, Noyce PR. Adolescent girls' use of the Internet for health information: issues beyond access. J Med Syst 2002 Dec;26(6):545-553. [Medline]
    18. Marshall LA, Williams D. Health information: does quality count for the consumer? Journal of Librarianship and Information Science 2016 Jun 30;38(3):141-156. [CrossRef]
    19. Anderson JG. Consumers of e-Health. Social Science Computer Review 2016 Aug 18;22(2):242-248. [CrossRef]
    20. Santer M, Muller I, Yardley L, Burgess H, Ersser SJ, Lewis-Jones S, et al. 'You don't know which bits to believe': qualitative study exploring carers' experiences of seeking information on the internet about childhood eczema. BMJ Open 2015;5(4):e006339 [FREE Full text] [CrossRef] [Medline]
    21. Silver MP. Patient perspectives on online health information and communication with doctors: a qualitative study of patients 50 years old and over. J Med Internet Res 2015 Jan;17(1):e19 [FREE Full text] [CrossRef] [Medline]
    22. Benotsch EG, Kalichman S, Weinhardt LS. HIV-AIDS patients' evaluation of health information on the internet: the digital divide and vulnerability to fraudulent claims. J Consult Clin Psychol 2004 Dec;72(6):1004-1011. [CrossRef] [Medline]
    23. Oh S, Worrall A. Health answer quality evaluation by librarians, nurses, and users in social Q&A. Library & Information Science Research 2013 Oct;35(4):288-298. [CrossRef]
    24. Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med 2011 Jul 19;155(2):97-107. [CrossRef] [Medline]
    25. Gryna F. Juran's quality control handbook. New York: McGraw-Hill College Division; 1988.
    26. Zhang Y, Wildemuth BM. Qualitative analysis of content. In: Wildemuth B, editor. Applications of Social Research Methods to Questions in Information and Library Science (2nd ed.). Westport, CT: Libraries Unlimited; 2017:318-329.
    27. Kim Y. Trust in health information websites: A systematic literature review on the antecedents of trust. Health Informatics J 2016 Jun;22(2):355-369. [CrossRef] [Medline]
    28. Fogg B, Soohoo C, Danielson D, Marable L, Stanford J, Tauber E. How Do Users Evaluate the Credibility of Web Sites?: A Study with over 2,500 Participants. 2003 Presented at: Proc Conf Des User Exp; June 06-07, 2003; San Francisco, California p. 1-15. [CrossRef]
    29. Robins D, Holmes J, Stansbury M. Consumer health information on the Web: The relationship of visual design and perceptions of credibility. J Am Soc Inf Sci 2009 Oct 02;61(1):13-29. [CrossRef]
    30. Eysenbach G, Köhler C. How do consumers search for and appraise health information on the world wide web? Qualitative study using focus groups, usability tests, and in-depth interviews. BMJ 2002 Mar 09;324(7337):573-577 [FREE Full text] [Medline]
    31. Metzger MJ, Flanagin AJ, Eyal K, Lemus DR, Mccann RM. Credibility for the 21st Century: Integrating Perspectives on Source, Message, and Media Credibility in the Contemporary Media Environment. Annals of the International Communication Association 2016 May 18;27(1):293-335. [CrossRef]
    32. Sillence E, Briggs P, Harris PR, Fishwick L. How do patients evaluate and make use of online health information? Soc Sci Med 2007 May;64(9):1853-1862. [CrossRef] [Medline]
    33. Hong T. The influence of structural and message features on Web site credibility. J Am Soc Inf Sci 2005 Jan 01;57(1):114-127. [CrossRef]
    34. Belkin NJ. Intelligent information retrieval: Whose intelligence? Intelligent information retrieval: Whose intelligence? Proc Fifth Int Symp Inf Sci Konstanz: Universitatsverlag Konstanz; 1996 Presented at: Proc Fifth Int Symp Inf Sci; 1996; Konstanz p. 25-31.
    35. Brunswik E. The Conceptual Framework of Psychology. Chicago, IL: University of Chicago Press; 1962.
    36. Wang P, Soergel D. A cognitive model of document use during a research project. Study I. Document selection. J Am Soc Inf Sci 1998 Feb;49(2):115-133. [CrossRef]
    37. Choi W, Stvilia B. Web credibility assessment: Conceptualization, operationalization, variability, and models. J Assn Inf Sci Tec 2015 May 13;66(12):2399-2414. [CrossRef]
    38. Henderson EM, Eccleston C. An online adolescent message board discussion about the internet: Use for pain. J Child Health Care 2015 Sep;19(3):412-418. [CrossRef] [Medline]
    39. Frisby G, Bessell TL, Borland R, Anderson JN. Smoking cessation and the Internet: a qualitative method examining online consumer behavior. J Med Internet Res 2002 Nov;4(2):E8 [FREE Full text] [CrossRef] [Medline]
    40. Peterson G, Aslani P, Williams KA. How do consumers search for and appraise information on medicines on the Internet? A qualitative study using focus groups. J Med Internet Res 2003 Dec 19;5(4):e33 [FREE Full text] [CrossRef] [Medline]
    41. Williams P, Nicholas D, Huntington P. Health information on the Internet: a qualitative study of NHS Direct Online users. Aslib Proceedings 2003 Dec;55(5/6):304-312. [CrossRef]
    42. Bernhardt J, Felter E. Online pediatric information seeking among mothers of young children: results from a qualitative study using focus groups. J Med Internet Res 2004 Mar 01;6(1):e7 [FREE Full text] [CrossRef] [Medline]
    43. Childs S. Developing health website quality assessment guidelines for the voluntary sector: outcomes from the Judge Project. Health Info Libr J 2004 Sep;21 Suppl 2:14-26 [FREE Full text] [CrossRef] [Medline]
    44. Adams S, de Bont A, Berg M. Looking for answers, constructing reliability: an exploration into how Dutch patients check web-based medical information. Int J Med Inform 2006 Jan;75(1):66-72. [CrossRef] [Medline]
    45. Crystal A, Greenberg J. Relevance criteria identified by health information users during Web searches. J Am Soc Inf Sci 2006 Aug;57(10):1368-1382. [CrossRef]
    46. Kerr C, Murray E, Stevenson F, Gore C, Nazareth I. Internet interventions for long-term conditions: patient and caregiver quality criteria. J Med Internet Res 2006 Jul 28;8(3):e13 [FREE Full text] [CrossRef] [Medline]
    47. Hoffman-Goetz L, Friedman DB. A qualitative study of Canadian Aboriginal women's beliefs about. J Cancer Educ 2007;22(2):124-128. [CrossRef] [Medline]
    48. Sillence E, Briggs P. Please advise: using the Internet for health and financial advice. Computers in Human Behavior 2007 Jan;23(1):727-748. [CrossRef]
    49. Sillence E, Briggs P, Harris P, Fishwick L. Health Websites that people can trust – the case of hypertension. Interacting with Computers 2007 Jan;19(1):32-42. [CrossRef]
    50. Buhi ER, Daley EM, Fuhrmann HJ, Smith SA. An observational study of how young people search for online sexual health information. J Am Coll Health 2009 Sep;58(2):101-111. [CrossRef] [Medline]
    51. Freeman KS, Spyridakis JH. Effect of Contact Information on the Credibility of Online Health Information. IEEE Trans Profess Commun 2009 Jun;52(2):152-166. [CrossRef]
    52. Mackert M, Kahlor L, Tyler D, Gustafson J. Designing e-health interventions for low-health-literate culturally diverse parents: addressing the obesity epidemic. Telemed J E Health 2009 Sep;15(7):672-677 [FREE Full text] [CrossRef] [Medline]
    53. Marton C. How Women With Mental Health Conditions Evaluate the Quality of Information on Mental Health Web sites: A Qualitative Approach. Journal of Hospital Librarianship 2010 Jul 28;10(3):235-250. [CrossRef]
    54. Kim H, Park S, Bozeman I. Online health information search and evaluation: observations and semi-structured interviews with college students and maternal health experts. Health Info Libr J 2011 Sep;28(3):188-199 [FREE Full text] [CrossRef] [Medline]
    55. Colombo C, Mosconi P, Confalonieri P, Baroni I, Traversa S, Hill SJ, et al. Web search behavior and information needs of people with multiple sclerosis: focus group study and analysis of online postings. Interact J Med Res 2014 Jul 24;3(3):e12 [FREE Full text] [CrossRef] [Medline]
    56. Lederman R, Fan H, Smith S, Chang S. Who can you trust? Credibility assessment in online health forums. Health Policy and Technology 2014 Mar;3(1):13-25. [CrossRef]
    57. McPherson AC, Gofine ML, Stinson J. Seeing is believing? A mixed-methods study exploring the quality and perceived trustworthiness of online information about chronic conditions aimed at children and young people. Health Commun 2014 Oct;29(5):473-482. [CrossRef] [Medline]
    58. Payton FC, Kvasny L, Kiwanuka-Tondo J. Online HIV prevention information: How black female college students are seeking and perceiving. Internet Research 2014 Jul 29;24(4):520-543. [CrossRef]
    59. Briones R. Harnessing the Web: How E-Health and E-Health Literacy Impact Young Adults' Perceptions of Online Health Information. Med 2 0 2015 Dec 31;4(2):e5 [FREE Full text] [CrossRef] [Medline]
    60. Rennis L, McNamara G, Seidel E, Shneyderman Y. Google it!: Urban community college students? use of the Internet to obtain self-care and personal health information. Coll Stud J 2015;49(3):414-426.
    61. Subramaniam M, St Jean B, Taylor NG, Kodama C, Follman R, Casciotti D. Bit by bit: using design-based research to improve the health literacy of adolescents. JMIR Res Protoc 2015 May 29;4(2):e62 [FREE Full text] [CrossRef] [Medline]
    62. Cunningham A, Johnson F. Exploring trust in online health information: a study of user experiences of patients.co.uk. Health Info Libr J 2016 Dec;33(4):323-328. [CrossRef] [Medline]
    63. Diviani N, van den Putte B, Meppelink CS, van Weert JC. Exploring the role of health literacy in the evaluation of online health information: Insights from a mixed-methods study. Patient Education and Counseling 2016 Jun;99(6):1017-1025. [CrossRef]
    64. Sillence E, Hardy C, Medeiros LC, LeJeune JT. Examining trust factors in online food risk information: The case of unpasteurized or 'raw' milk. Appetite 2016 Apr 01;99:200-210. [CrossRef] [Medline]
    65. Alsem MW, Ausems F, Verhoef M, Jongmans MJ, Meily-Visser JMA, Ketelaar M. Information seeking by parents of children with physical disabilities: An exploratory qualitative study. Res Dev Disabil 2017 Jan;60:125-134. [CrossRef] [Medline]
    66. Champlin S, Mackert M, Glowacki EM, Donovan EE. Toward a Better Understanding of Patient Health Literacy: A Focus on the Skills Patients Need to Find Health Information. Qual Health Res 2017 Jul;27(8):1160-1176. [CrossRef] [Medline]
    67. Cusack L, Desha LN, Del Mar CB, Hoffmann TC. A qualitative study exploring high school students' understanding of, and attitudes towards, health information and claims. Health Expect 2017 Dec;20(5):1163-1171 [FREE Full text] [CrossRef] [Medline]
    68. Peddie KA, Kelly-Campbell RJ. How people with hearing impairment in New Zealand use the Internet to obtain information about their hearing health. Computers in Human Behavior 2017 Aug;73:141-151. [CrossRef]
    69. Scantlebury A, Booth A, Hanley B. Experiences, practices and barriers to accessing health information: A qualitative study. Int J Med Inform 2017 Dec;103:103-108. [CrossRef] [Medline]
    70. Rieh SY. Judgment of information quality and cognitive authority in the Web. J Am Soc Inf Sci 2002;53(2):145-161. [CrossRef]
    71. Bates BR, Romina S, Ahmed R, Hopson D. The effect of source credibility on consumers' perceptions of the quality of health information on the Internet. Med Inform Internet Med 2006 Mar;31(1):45-52. [CrossRef] [Medline]
    72. Eastin M. Credibility Assessments of Online Health Information: The Effects of Source Expertise and Knowledge of Content. J Comput-Mediat Commun 2006 Jun 23;6(4). [CrossRef]
    73. Hovland C, Janis I, Kelley H. Communication and persuasion; psychological studies of opinion change. New Haven, CT: Yale University Press; 1953.
    74. Burkell J. Health Information Seals of Approval: What do they Signify? Information, Communication & Society 2004 Jan;7(4):491-509. [CrossRef]
    75. Allam A, Sak G, Diviani N, Schulz PJ. Do quality markers for health websites affect the perception of vaccination webpages? Computers in Human Behavior 2017 Feb;67:273-281. [CrossRef]


    Abbreviations

    CDC: Centers for Disease Control and Prevention
    CINAHL: Cumulative Index to Nursing and Allied Health
    NIH: National Institutes of Health


    Edited by G Eysenbach; submitted 19.10.18; peer-reviewed by L Sbaffi, A Jadhav, B van den Putte; comments to author 07.02.19; revised version received 19.03.19; accepted 08.04.19; published 02.05.19

    ©Yalin Sun, Yan Zhang, Jacek Gwizdka, Ciaran B. Trace. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 02.05.2019.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.