JMIR Publications

Journal of Medical Internet Research

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 07.05.15 in Vol 17, No 5 (2015): May

This paper is in the following e-collection/theme issue:

    Review

    Low Health Literacy and Evaluation of Online Health Information: A Systematic Review of the Literature

    1Amsterdam School of Communication Research / ASCoR, Department of Communication Science, University of Amsterdam, Amsterdam, Netherlands

    2Trimbos Institute, Netherlands Institute for Mental Health and Addiction, Utrecht, Netherlands

    3University Library, University of Amsterdam, Amsterdam, Netherlands

    Corresponding Author:

    Nicola Diviani, PhD

    Amsterdam School of Communication Research / ASCoR

    Department of Communication Science

    University of Amsterdam

    P.O. Box 15791

    Amsterdam, 1001 NG

    Netherlands

    Phone: 31 6 15254105

    Fax:31 20 5253681

    Email:


    ABSTRACT

    Background: Recent years have witnessed a dramatic increase in consumer online health information seeking. The quality of online health information, however, remains questionable. The issue of information evaluation has become a hot topic, leading to the development of guidelines and checklists to design high-quality online health information. However, little attention has been devoted to how consumers, in particular people with low health literacy, evaluate online health information.

    Objective: The main aim of this study was to review existing evidence on the association between low health literacy and (1) people’s ability to evaluate online health information, (2) perceived quality of online health information, (3) trust in online health information, and (4) use of evaluation criteria for online health information.

    Methods: Five academic databases (MEDLINE, PsycINFO, Web of Science, CINAHL, and Communication and Mass-media Complete) were systematically searched. We included peer-reviewed publications investigating differences in the evaluation of online information between people with different health literacy levels.

    Results: After abstract and full-text screening, 38 articles were included in the review. Only four studies investigated the specific role of low health literacy in the evaluation of online health information. The other studies examined the association between educational level or other skills-based proxies for health literacy, such as general literacy, and outcomes. Results indicate that low health literacy (and related skills) are negatively related to the ability to evaluate online health information and trust in online health information. Evidence on the association with perceived quality of online health information and use of evaluation criteria is inconclusive.

    Conclusions: The findings indicate that low health literacy (and related skills) play a role in the evaluation of online health information. This topic is therefore worth more scholarly attention. Based on the results of this review, future research in this field should (1) specifically focus on health literacy, (2) devote more attention to the identification of the different criteria people use to evaluate online health information, (3) develop shared definitions and measures for the most commonly used outcomes in the field of evaluation of online health information, and (4) assess the relationship between the different evaluative dimensions and the role played by health literacy in shaping their interplay.

    J Med Internet Res 2015;17(5):e112

    doi:10.2196/jmir.4018

    KEYWORDS



    Introduction

    Recent years have witnessed a dramatic increase in consumer online health information seeking [1]. Online health information deserves particular attention because studies on the content of health-related websites have highlighted inaccuracies that raise concerns about the quality of the online health information encountered by consumers [2-6]. The limited accuracy of information is often the result of one of the distinctive features of the Internet, that is, that anyone can potentially publish health-related information. Online health information seeking thus poses several major challenges to health information users, as it requires them to undertake an active role in evaluating a vast amount of often unverified health information on the Internet [7]. As a result, people experiencing difficulties evaluating online health information may be exposed to wrong or incomplete information, which has been shown to be related to adverse health outcomes, such as low participation in screening programs or low adherence to treatments [8]. Evidently, more attention needs to be given to the issue of quality of online health information and in particular to people’s ability to evaluate it [2-6].

    Several guidelines and checklists to improve the quality of online health information have been developed, for example by the Standford Persuasive Tech Lab, the Health On the Net Foundation (HONcode), Web Médica Acreditada, and Centrale santé (Netscoring criteria) (see Kim et al [9] for a summary view). These tools can be useful for Web designers and providers of health information to develop high-quality health websites. At the same time, the guidelines could be used by users as evaluation criteria to assess online health information. However, these criteria are likely known and adopted only by specific segments of the population, resulting in disparities in people’s ability to evaluate online health information. The knowledge gap hypothesis, for instance, states that as a result of increasing mass media exposure, individuals in the higher socioeconomic strata of society tend to acquire information faster than people in lower ones. So the gap in knowledge between the two tends to increase rather than decrease [10]. It is likely that traditionally disadvantaged groups—such as those with lower education or lower health literacy—will be the ones at higher risk for disparities in this context [11].

    Among the determinants of health disparities, people’s health literacy has been proven to play a crucial role in the context of health information seeking. Health literacy has originally been defined as “the cognitive and social skills which determine the motivation and ability of individuals to gain access, to understand, and use information in ways which promote and maintain good health” [12]. A closer look at the different conceptualizations of health literacy proposed in the last years (eg, [12-19]) shows that almost all of them explicitly or implicitly include people’s ability to deal with (ie, obtain, process, evaluate, and use) health information among the skills a person should possess in order to be considered health literate. Several studies have provided evidence of differences in how people with different levels of health literacy seek, find, understand, and use online health information. For instance, low health literate people have been shown to search less for health information, choose different information sources, and have a poorer ability to interpret medication labels or health messages [20-23]. In contrast, little attention has been devoted to how consumers—and in particular those with low health literacy—evaluate online health information [24]. Additionally, to date, no studies have systematically summarized existing evidence on the role of health literacy in the context of the evaluation of online health information.

    The main objective of this study was to address this research gap and provide a comprehensive description of how low health literacy impacts people’s evaluation of online health information. This translates to four distinct research questions aimed at understanding whether and how people’s health literacy is related to their ability to evaluate online health information (RQ1), perceived quality of online health information (RQ2), people’s trust in the Internet as a source of online health information (RQ3), and the use of evaluation criteria for online health information (RQ4).

    The rationale behind RQ1 is the fact that a relationship between health literacy and ability to evaluate the quality of health information has been explicitly or implicitly suggested by several health literacy conceptualizations [12-19] but has not been systematically verified so far.

    Information evaluation, however, does not depend only on the characteristics of the audience (in our case, people’s ability to evaluate online health information). As acknowledged already at the very beginning of scholarly interest in the field of credibility, characteristics of the message and the source can play a role as well [25-28]. This is the reason why RQ2 is about the relationship between health literacy and perceived quality of online health information (message level). Perceived information quality is a multifaceted concept encompassing several dimensions, which in turn can be grouped in different categories [29,30]. For the purposes of this review, we will focus on the dimensions of perceived reliability and accuracy, which Wang and Strong [31] have defined as intrinsic information quality.

    It has been found that perceived information quality does not necessarily imply intention to rely on it [32]. Someone could, for instance, perceive a message as being of high quality but not trust the source because of external factors (such as previous negative experiences) and thus decide not to act on the information. Therefore, RQ3 concerns overall trust in the Internet as a source of health information. In the context of computer-mediated communication, trust has been defined in terms of dependability and is a subjective judgment about whether a person (or, in the case of online health information, a digital object) is worth being relied on [32,33].

    Last, as past research in other fields has shown that people use several different criteria when evaluating online information, including relying on formal (eg, color of the webpage) or contextual (eg, position in Google search results) aspects of the website, or evaluating the information based on previous knowledge [34], RQ4 is aimed at understanding whether health literacy plays a role in the choice and use of these evaluation criteria.

    Answers to these four questions will provide us with a new and more comprehensive understanding of the role played by low health literacy in the evaluation of online information. At the same time, the results of the review will allow us to identify possible areas where consumer education could have an impact on improving low health literate people’s ability to correctly evaluate online health information.


    Methods

    Data Sources and Search Strategy

    During the third week of January 2014, five academic databases from different relevant disciplines (Medline, PsycINFO, CINAHL, Web of Science, and Communication and Mass Media Complete) were systematically searched for peer-reviewed literature describing consumers’ evaluation of online health information. No time limits were set because the topic of online health information seeking is relatively recent. Search terms used included a combination of Medical Subject Headings (MeSH) terms and free terms covering the four domains, “online information”, “health”, “evaluation”, and “health literacy”, combined with the Boolean operator AND (see Multimedia Appendix 1). A preliminary search showed that the number of articles explicitly mentioning health literacy was limited, so the search was modified to include some of its most common proxies or indicators. Since health literacy has been defined as a set of skills [12], only skills-based indicators (eg, educational attainment, reading ability, or general literacy) were included in the search. These indicators can be considered as proxies of health literacy because of their conceptual similarities (eg, they all refer to teachable skills) and the existence of a direct link between the concepts. Other common, mainly sociodemographic, indicators (eg, age, income, or ethnicity) were excluded because of the more complex nature of their relationship with health literacy. The original search strategy was developed for PsycINFO and subsequently adapted to the peculiarities and requirements of the other databases. These terms were included in the original search, allowing us to refine it. References cited in included articles were reviewed manually, and a Google Scholar and Web of Science search for recent articles citing the ones included in the review was performed in order to identify further additional articles relevant to this review (snowball method).

    Study Inclusion/Exclusion Criteria

    A two-phase screening process was conducted. After de-duplication, the first author (ND) screened titles and abstracts of all retrieved articles in order to identify possible relevant articles (first phase). Abstracts were selected for full text screening (second phase) if they (1) were written in English, (2) reported original results, qualitative or quantitative, (3) studied consumer online health information, (4) mentioned evaluation of the information by consumers/patients, and (5) had been conducted in a low health literacy population (or in a sample of the above-described proxies of low health literacy) OR subgroup analyses were conducted in a sample of low health literate people (or proxies of low health literacy). Excluded were non-empirical articles (such as reviews, commentaries, or editorials), articles describing empirical studies conducted among health care providers, content analyses of websites, quality assessments of websites, and articles reporting on research conducted in samples that were not explicitly described as low health literate (or proxies) and did not present subgroup analyses for the low health literacy (or proxies) group. No selection based on the country where the study was conducted was made, and the same inclusion criteria were used for each country. In order to estimate reliability of the screening process, 10% of the abstracts and all the selected full texts were independently assessed by a second researcher. Initial intercoder agreement (Cohen’s kappa >.70 for both title/abstract and full-text screening) was substantial [35], and all disagreements regarding full texts were resolved during consensus meetings that were held on a regular basis during the whole screening process.

    Data Extraction

    Overview

    Besides the formal characteristics of the included papers—author(s), publication date, study design, study population, and samplingdata were extracted from all articles on the following aspects, which were deemed relevant to answer our research questions.

    Predictors

    Data were extracted about the predictors used in the study and the measures used to assess them. The main predictor of interest was health literacy, which can be assessed using different tools, for example, the Rapid Estimate of Adult Literacy in Medicine (REALM) [36], the Short Test of Functional Health Literacy in Adults (S-TOFHLA) [37,38], or the Newest Vital Signs (NVS) [39]. As mentioned earlier, however, included studies could also describe differences in one of the outcomes of interest related to differences in education or other skills-based proxies for health literacy [40]. Each study could address one or more predictors.

    Outcomes

    Data were extracted on the outcomes addressed in the studies and the measures used to assess them. The four outcomes of interest were (1) ability to evaluate online health information, (2) perceived quality of online health information, (3) trust in the Internet as a source of online health information, and (4) use of evaluation criteria for online health information. Each study could address one or more outcomes.

    Association Between Predictors and Outcomes

    All qualitative or quantitative evidence of (or lack of) an association between one of the predictors and one of the four outcomes of interest were extracted.

    Data Synthesis

    Given the heterogeneity of study designs, samples, predictors, and outcome measures in our articles pool, results could not be synthetized quantitatively using meta-analytic techniques. The findings were therefore synthetized narratively and structured according to the different outcomes under investigation.


    Results

    Included Studies

    The initial search resulted in 17,507 articles. In the process of reviewing articles identified through the initial searches, 3 additional articles were identified through cited references, bringing the total to 17,510 articles. After duplicates were removed, the remaining number of articles was 13,632, of which 13,378 were discarded after reviewing titles and abstract. An additional 216 articles were discarded after reviewing the articles in their entirety, resulting in 38 articles [24,41-77]. The whole process is illustrated in detail by the flow diagram in Figure 1.

    Figure 1. Flow diagram of the screening process.
    View this figure

    Characteristics of Included Studies

    The basic characteristics of the final pool of articles included in the systematic review are described in Table 1 [41-77]. The 38 studies were published between 2001 and 2013. Most of them were conducted in North America (24/38; 63%), five were conducted in Europe (13%), four (11%) in Asia, four (11%) in Australia, and one in Africa (3%). Study populations varied widely, ranging from the general population to specific patient groups, and so did sample sizes, ranging from N=8 up to N=8586. All studies were non-experimental, with the vast majority being cross-sectional surveys (35/38; 92%), and the remaining were qualitative studies (1 focus group study [60] and 2 qualitative observational studies [45,50]).

    According to commonly used approaches for rating the quality of evidence in systematic reviews (eg, Grades of Recommendation, Assessment, Development and Evaluation [GRADE]; see [78]), the quality level of the evidence has to be considered low because all the studies included in this review are non-interventional in nature.

    Table 1. Characteristics of included studies.
    View this table

    Predictors Included in the Studies

    Only four of the included studies (4/38, 11%) specifically described the relationship between health literacy and one or more of the four outcomes of interest [45,52,60,76]. Health literacy was measured using different instruments: the NVS [52,76], the TOFHLA [43], and S-TOFHLA [60].

    The majority of the included studies (33/38, 87%) described the relationship between educational level and one or more of the outcomes. In most of them, educational level was operationalized either as number of years of education or as the highest achieved degree. In some cases, however, other operational definitions were used. One study compared a university sample with a mid-high socioeconomic and a low socioeconomic sample [48], two compared different ethnic groups with different educational levels [66,75], one compared in-school versus out-of-school young people [67], and one last study compared students in different grades and enrolled in programs with or without a health focus within the same school [52].

    Five studies (13%) described the relationship between other skills-based proxies for health literacy and one or more of the outcomes. These skills included reading comprehension [43], comfort speaking English [47], general literacy [45], and ability to understand health information [24]. Although it could be argued that this last skill is conceptually very similar to health literacy, the authors did not define it as such in their paper. In addition, health literacy was not measured with a recognized measure but by a single item. In the fifth study, a distinction was made between skilled (younger than 30 years, higher educational level and more experienced using the Web) and less-skilled (50 or older) participants [50]. As some studies reported results related to more than one predictor, percentages add up to more than 100%, and studies might be referred to more than once throughout the results.

    Outcomes

    Overview

    The majority of the studies included in the review included trust in online health information as an outcome variable (53%, 20/38). The second most common outcome was evaluation ability, included in 26% of the studies (10/38), followed by perceived quality of online health information (defined in the studies as perceived credibility, reliability, accuracy, and worth), which was included in 21% of the studies (8/38). The least frequent outcome variable in the pool of articles was the use of evaluation criteria, which was included in 5 studies only (14%). As some studies reported results on more than one outcome, percentages add up to more than 100%, and studies might be referred to more than once throughout the results.

    Ability to Evaluate Online Health Information

    Our first research question focused on the relationship between health literacy and ability to evaluate online health information. This specific aspect was addressed in 10 (26%) of the 38 studies (see Table 2 [43,45,52,56-58,64,65,72,73]). Evaluation ability was mostly assessed using self-report measures. Five studies [52,57,58,65,72] used the eHealth Literacy scale (eHEALS)—a measure assessing several aspects related to online health information seeking, including people’s perceived ability to evaluate online health resources and to distinguish high quality from low quality online health information—or some of its items [79]. Other studies asked the respondents to self-assess their ability to evaluate online health information by means of a single item measure [64] or qualitatively by means of an open question [45]. Among the studies using objective measures, van Deursen and van Dijk [73] asked their participants to perform an evaluation task derived from the eHEALS. In the other studies, the participants were asked to evaluate websites of varying quality [43,56].

    Two studies assessed the role of health literacy in people’s ability to evaluate online health information. The first study showed that low health literacy was associated with lower eHEALS scores [52], while in the second study low health literacy was shown to be associated with lower quality ratings of a high-quality website and higher quality ratings of a low-quality website [43].

    Six out of the nine studies focusing on educational level showed a positive relationship between educational attainment and perceived or actual ability to evaluate online health information [43,52,57,58,65,73]. The studies by Murray et al [64] and van der Vaart et al [72] did not find any significant difference among different education groups. A last study found that lower education was associated with assigning a higher quality rating to a low-quality website but did not find any association between education and evaluation of a high-quality website [56].

    Last, two studies reported on ability to evaluate online health information in relation to other skills. In a small qualitative study conducted in a sample with low general literacy, 7 out of the 8 respondents reported finding it very easy to locate trustworthy health information on the Internet [45]. The second study identified a positive association between reading ability and correct evaluation of the quality of a high-quality and a low-quality website [43].

    Perceived Quality of Online Health Information

    Eight studies (21%) reported on perceived quality of online health information. Except in one case where a multidimensional scale was used [46], this aspect was measured by means of single-item measures. Most studies did not refer directly to information quality but to one of its dimensions, that is, reliability [51,69,74], perceived accuracy [44,53,67], and perceived worth [46] (see Table 3 [44,46,50,51,53,67,69,74]). No studies on perceived information quality included health literacy as a predictor.

    The majority of the studies focusing on educational level failed to find significant associations between education and perceived quality of online health information. One study found a positive association [53], and one a negative association [44]. Another study found contrasting results: while out-of-school adolescents tended to describe online health information as more accurate than in-school adolescents, the latter group assigned higher overall quality to online health information as compared to out-of-school adolescents [67].

    Only one study reported on the relationship between skills-based proxies for health literacy and perceived quality of online health information. The study, however, did not find any differences among skilled and less-skilled participants: both groups doubted the quality of online health information [50].

    Table 2. Outcome 1: Ability to evaluate online health information.
    View this table
    Table 3. Outcome 2: Perceived quality of online health information.
    View this table
    Trust in Online Health Information

    Trust in online health information was the outcome measure of more than half of the studies included in this review (20/38, 53%) (Table 4 [24,41,47,48,54,55,59,61,63,66-71,74-76]). Among those, three studies asked the participants to rate their trust in online health information on specific health topics (cancer and nutrition [54,66,76]). All other studies referred to general trust in online health information and assessed it by means of a single-item measure.

    Only one study investigated trust in online health information in relation to health literacy and did not find any significant relationship [76].

    Ten out of the 17 studies reporting on the relationship between educational level and trust in online health information identified a positive association. Among the other studies, two found a negative association [59,66] and four found no significant association [24,48,69,74] between educational level and trust. One study reported contrasting results: Maguire et al [61] found a positive association between education and trust among respondents with schizophrenia but not among non-schizophrenia respondents.

    Two studies reported on the relationship between skills-based proxies for health literacy and trust in online health information. A positive association between comfort speaking English [47] and ease in understanding health information [24] and trust was found.

    Table 4. Outcome 3: Trust in online health information.
    View this table
    Use of Evaluation Criteria

    Five studies (14%) investigated the last outcome of interest, namely people’s use of evaluation criteria for online health information (Table 5 [42,49-51,60]).

    Only the study by Mackert et al [60] was about health literacy. This study, conducted in a low health literate-only sample, showed that low health literates used position in search results, quality of pictures, celebrity endorsement, and website authorship as criteria to evaluate online health information. With regard to website authorship, the study showed that almost all participants lacked trust in government or religious authorities as sources of online health information, whereas university researchers were generally considered trusted information providers.

    Reliance on website authorship as an evaluation criterion was also found in one of the studies investigating the role of educational level. In his study, Dutta-Bergman [49] showed that educational level was positively associated with trust in information coming from medical universities and federal institutions and negatively associated with trust in information from health insurance companies. No differences related to education were found with regard to trust in online health information coming from the local doctor. Bates et al [42] showed that there was no consistent relationship between educational level and using readability of websites as a criterion to evaluate website quality. This finding suggests that ease of reading is not widely used as a criterion to evaluate the quality of the websites. Gauld and Williams [51] found that respondents with higher educational level were more likely to check credentials (eg, name or qualifications of the author) when evaluating health websites.

    With regard to the role played by other skills-based proxies, Feufel and Stahl [50] showed in their study among skilled and less-skilled people that consistency with search intentions was used as a criterion to evaluate online health information. However, consistency referred to different things in the two groups: for the majority of less-skilled participants, this meant that a website confirmed a priori opinions, while for the majority of skilled participants, this meant that a website yielded the information that was searched for.

    Table 5. Outcome 4: Use of evaluation criteria.
    View this table

    Discussion

    Principal Findings

    People’s health literacy is deemed to play an important role in the context of online health information seeking because according to most definitions it includes the ability to evaluate health information from different sources [12-19]. One of the main aims of this review was to identify and systematically summarize existing literature in order to collect evidence on the effect of low health literacy on the evaluation of online health information. The review provided us with indications of an overall (positive) association between health literacy (or one of its skills-based proxies) and both people’s ability to evaluate online health information (RQ1) and trust in the Internet as a source of health information (RQ3). On the other hand, evidence on the association between health literacy and both perceived quality of online health information (RQ2) and people’s use of evaluation criteria for online health information (RQ4) was inconsistent.

    The limited number and the heterogeneity of studies using health literacy as a predictor makes it hard to get a clear picture of how people’s health literacy levels impact the evaluation of online health information. However, the included studies give us some indications that low health literacy may have a negative impact. Indeed, low health literate individuals use evaluation criteria that do not correspond to the well-established quality criteria [9]. To illustrate, one study reports that they do not trust online health information from the government or that they use the position of a website in search results or the quality of images to evaluate the quality of online health information [60], whereas evidence shows that information provided from institutional sources is usually accurate [80,81] and that position in search results and image quality are not among the criteria that should be used to judge the quality of a website [9]. This could at least partly explain why, as reported in another study [43], low health literate respondents, compared to their high health literate counterparts, have been shown to give higher-quality ratings to low-quality websites, and lower ratings to high-quality websites. At the same time, it has also been found that low health literacy is correlated with lower eHEALS scores (which include people’s perceived ability to evaluate online health information). Confidence in one’s own information skills has been shown to be positively related to information use [82].

    What was observed in the few studies about health literacy was mostly confirmed in the higher amount of studies investigating differences in the evaluation of online health information related to educational level or other health literacy-related skills. Although we recognize that the diversity of measures used and the reliance on dichotomous measures of educational level may limit the accuracy of the results [40], this gives us more confidence in our findings. Included studies have shown that, overall, individuals with lower educational levels have worse actual and self-rated skills to evaluate the quality of online health information and lower trust in online health information compared to their more educated counterparts. Regarding perceived quality of online health information or people’s use of evaluation criteria, the limited number of studies and the diversity of samples and measures, however, does not allow us to draw conclusions about the impact of educational level or other skills-based proxies of health literacy, leaving two of the main research questions of this study mainly unanswered.

    Besides providing us with indications on the role played by people’s health literacy in their evaluation of online health information, our review also highlighted some important gaps and limitations of research in this field. The main gap identified is probably the fact that, despite the undeniable theoretical importance of health literacy for online health information seeking, only four studies have specifically investigated the association between health literacy and the evaluation of online health information. Moreover, one of these studies was conducted in a low health literacy-only sample [60], not allowing comparisons across different health literacy levels. All the other studies in our review compared people with different educational levels or different levels of other literacy-related skills.

    A second important limitation of current research highlighted by the review is the lack of shared definitions and measures. This has been shown to be the case not only for health literacy but also for other more commonly used predictors such as educational level, making comparisons across studies and summaries of existing evidence almost impossible. Additionally, only a few studies measured actual online health information evaluation skills, asking their participants to perform actual evaluation tasks. All the other studies relied on the participants’ self-rated ability. As shown in the study by van der Vaart et al [83], who compared their respondents’ eHEALS scores with an Internet performance test, self-rated ability does not seem to adequately capture people’s actual skills. Further research efforts should thus be devoted to the development and validation of a shared measure of online health information evaluation skills that is better able to reflect people’s real ability in this context.

    Last, among the studies investigating more than one outcome variable, no information was found on the relationship between them. For this reason, no conclusions can be drawn on the interplay among the different evaluative dimensions.

    Limitations

    This review has several limitations. The first limitation consists of the limited number of studies using health literacy as a predictor. Despite the fact that education has been shown to be related to (and it is often used as a proxy for) health literacy (eg, see [84]), this fact limits the extent to which our results can be generalized. A second important limitation of this review is the fact that all the included studies were non-interventional, with the consequence that the overall quality of the evidence has to be considered low [78]. Although causality is not an issue in this context (information evaluation cannot influence people’s literacy level), the uncontrolled nature of cross-sectional studies might fail to account for alternative explanations for the phenomenon under investigation [85]. This adds to the limited generalizability of the findings of this review. Finally, due to the heterogeneity of samples and outcome measures, it was not possible to conduct a meta-synthesis, with the result that no quantitative summary of the evidence could be provided.

    Conclusions and Future Directions

    Despite its limitations, this systematic review on the role of health literacy in the evaluation of online health information has provided us with important insights on this topic. These insights have allowed us to draw some preliminary conclusions and, most importantly, have highlighted the main outcomes and limitations of current research in this relatively new and unexplored field.

    From a research perspective, our findings are to be considered an indication of the fact that health literacy indeed plays a role in the evaluation of online health information and thus this topic is worth more scholarly attention. Based on the results of this review, the future research agenda in this field should include (1) a specific focus on health literacy, (2) more attention to the identification of the different criteria people use to evaluate online health information, (3) the development of shared definitions and measures for the most commonly used outcomes in the field of evaluation of online health information, and (4) the assessment of the relationship between the different evaluative dimensions and the role played by health literacy in shaping their interplay. Only by first addressing these research gaps will it be possible, in line with what has been called for in a recent Cochrane review [86], to develop high-quality interventions to enhance low health literacy individuals’ ability to appraise online health information as well as to develop well-designed randomized controlled trials to investigate their effects. A better understanding of how people appraise online health information is also crucial in view of the investigation of the impact of their information evaluation ability on their interactions with health care providers and ultimately on health outcomes.

    From a practice perspective, the results of this review should be sufficient to urge public health officials and health care providers to start devoting particular attention to the online health information seeking behavior of low health literate citizens and provide them with targeted advice on criteria to correctly assess the quality of the information they find online.

    Acknowledgments

    The authors wish to thank Anne Kimman for her valuable help in the abstract and full-text review process. This study was funded by a personal grant awarded to ND by the Swiss National Science Foundation (grant number P2TIP1_148345). The funding source had no role in study design, data collection, analysis, interpretation, or writing of the report.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Search strategy.

    PDF File (Adobe PDF File), 34KB

    References

    1. Fox S, Duggan M. Health Online 2013. 2013.   URL: http://pewinternet.org/Reports/2013/Health-online.aspx [accessed 2015-04-25] [WebCite Cache]
    2. Berland GK, Elliott MN, Morales LS, Algazy JI, Kravitz RL, Broder MS, et al. Health information on the Internet: accessibility, quality, and readability in English and Spanish. JAMA 2001;285(20):2612-2621 [FREE Full text] [Medline]
    3. Bernstam EV, Walji MF, Sagaram S, Sagaram D, Johnson CW, Meric-Bernstam F. Commonly cited website quality criteria are not effective at identifying inaccurate online information about breast cancer. Cancer 2008 Mar 15;112(6):1206-1213 [FREE Full text] [CrossRef] [Medline]
    4. Eysenbach G, Powell J, Kuss O, Sa ER. Empirical studies assessing the quality of health information for consumers on the world wide web: a systematic review. JAMA 2002;287(20):2691-2700. [Medline]
    5. Hardey M. 'E-health': the internet and the transformation of patients into consumers and producers of health knowledge. Info, Comm & Soc 2001 Oct;4(3):388-405. [CrossRef]
    6. Stvilia B, Mon L, Yi YJ. A model for online consumer health information quality. J Am Soc Inf Sci Technol 2009;60(9):1781-1791. [CrossRef]
    7. Flanagin AJ, Metzger MJ. Digital Media and Youth: Unparalleled Opportunity and Unprecedented Responsibility. In: Metzer MJ, Flanagin AJ, editors. Digital Media, Youth,Credibility. The John D.Catherine T. MacArthur Foundation Series on Digital Media and Learning. Cambridge, MA: The MIT Press; 2008:5.
    8. Cline RJ, Haynes KM. Consumer health information seeking on the Internet: the state of the art. Health Educ Res 2001 Dec;16(6):671-692 [FREE Full text] [Medline]
    9. Kim P, Eng TR, Deering MJ, Maxfield A. Published criteria for evaluating health related web sites: review. BMJ 1999 Mar 6;318(7184):647-649 [FREE Full text] [Medline]
    10. Tichenor PJ, Donohue GA, Olien CN. Mass media flow and differential growth in knowledge. Public Opin Q 1970;34(2):159-170. [CrossRef]
    11. Viswanath K, Kreuter MW. Health disparities, communication inequalities, and eHealth. Am J Prev Med 2007 May;32(5 Suppl):S131-S133 [FREE Full text] [CrossRef] [Medline]
    12. Nutbeam D. Health promotion glossary. Health Promot 1986 May;1(1):113-127. [Medline]
    13. Ishikawa H, Yano E. Patient health literacy and participation in the health-care process. Health Expect 2008 Jun;11(2):113-122. [CrossRef] [Medline]
    14. Jordan JE, Buchbinder R, Osborne RH. Conceptualising health literacy from the patient perspective. Patient Educ Couns 2010 Apr;79(1):36-42. [CrossRef] [Medline]
    15. Kickbusch I. Health literacy: a search for new categories. Health Promot Int 2002 Mar;17(1):1-2 [FREE Full text] [Medline]
    16. Nutbeam D. Health literacy as a public health goal: A challenge for contemporary health education and communication strategies into the 21st century. Health Promot Internation 2000;15(3):259-267. [CrossRef]
    17. Schulz P, Nakamoto K. Emerging themes in health literacy. Stud Commun Sci 2005;5(2):1-10.
    18. Schulz PJ, Nakamoto K. Health literacy and patient empowerment in health communication: the importance of separating conjoined twins. Patient Educ Couns 2013 Jan;90(1):4-11. [CrossRef] [Medline]
    19. Zarcadoolas C, Pleasant A, Greer DS. Understanding health literacy: An expanded model. Health Promot Internat 2005 Jun;20(2):195-203. [CrossRef]
    20. Anker AE, Reinhart AM, Feeley TH. Health information seeking: a review of measures and methods. Patient Educ Couns 2011 Mar;82(3):346-354. [CrossRef] [Medline]
    21. Hibbard JH, Tusler M. How much do health literacy and patient activation contribute to older adults' ability to manage their health?. Washington, DC: AARP Public Policy Institute; 2005.
    22. Reyna VF, Nelson WL, Han PK, Dieckmann NF. How numeracy influences risk comprehension and medical decision making. Psychol Bull 2009 Nov;135(6):943-973 [FREE Full text] [CrossRef] [Medline]
    23. von WC, Semmler C, Good A, Wardle J. Health literacy and self-efficacy for participating in colorectal cancer screening: The role of information processing. Patient Educ Couns 2009 Jun;75(3):352-357. [CrossRef] [Medline]
    24. Ye Y. Correlates of consumer trust in online health information: findings from the health information national trends survey. J Health Commun 2011 Jan;16(1):34-49. [CrossRef] [Medline]
    25. Hovland CI. Changes in attitude through communication. The Journal of Abnormal and Social Psychology 1951;46(3):424-437. [CrossRef]
    26. Hovland C, Janis I, Kelley H. Communication and persuasion. New Haven, CT: Yale University Press; 1953.
    27. Hovland CI, Lumsdaine CI, Sheffield FD. Experiments on mass communication. Princeton, NJ: Princeton University Press; 1949.
    28. Hovland CI, Weiss W. The influence of source credibility on communication effectiveness. Public Opin Q 1951;15(4):635-650. [CrossRef]
    29. Knight S, Burn JM. Developing a framework for assessing information quality on the world wide web. Informing Science: International Journal of an Emerging Transdiscipline 2005;8(5):159-172.
    30. Rieh SY. Judgment of information quality and cognitive authority in the Web. J Am Soc Inf Sci 2002;53(2):145-161. [CrossRef]
    31. Wang RY, Strong DM. Beyond accuracy: What data quality means to data consumers. J Manage Inf Syst 1996;12(4):5-33.
    32. Fogg B, Tseng H. The elements of computer credibility. 1999 Presented at: SIGCHI conference on human factors in computing systems; 1999; Pittsburgh, PA.
    33. Shneiderman B. Designing trust into online experiences. Commun. ACM 2000;43(12):57-59. [CrossRef]
    34. Fogg B, Marshall J, Laraki O, Osipovich A, Varma C, Fang N. What makes web sites credible? A report on a large quantitative study. 2001 Presented at: SIGCHI conference on human factors in computing systems; 2001; Seattle, WA.
    35. Landis JR, Koch GG. The Measurement of Observer Agreement for Categorical Data. Biometrics 1977 Mar;33(1):159. [CrossRef]
    36. Davis TC, Long SW, Jackson RH, Mayeaux EJ, George RB, Murphy PW, et al. Rapid estimate of adult literacy in medicine: a shortened screening instrument. Fam Med 1993 Jun;25(6):391-395. [Medline]
    37. Parker RM, Baker DW, Williams MV, Nurss JR. The test of functional health literacy in adults: a new instrument for measuring patients' literacy skills. J Gen Intern Med 1995 Oct;10(10):537-541. [Medline]
    38. Nurss J, Parker R, Williams M, Baker D. Directions for administration and scoring and technical data, short test of functional health literacy in adults (STOFHLA-English and STOFHLA-Spanish). Atlanta, GA: Center for the Study of Adult Literacy; 1998.
    39. Weiss BD, Mays MZ, Martz W, Castro KM, DeWalt DA, Pignone MP, et al. Quick assessment of literacy in primary care: the newest vital sign. Ann Fam Med 2005;3(6):514-522 [FREE Full text] [CrossRef] [Medline]
    40. Baker DW. The meaning and the measure of health literacy. J Gen Intern Med 2006 Aug;21(8):878-883 [FREE Full text] [CrossRef] [Medline]
    41. AlGhamdi KM, Moussa NA. Internet use by the public to search for health-related information. Int J Med Inform 2012 Jun;81(6):363-373. [CrossRef] [Medline]
    42. Bates BR, Romina SM, Ahmed R. The effect of improved readability scores on consumers' perceptions of the quality of health information on the internet. J Cancer Educ 2007;22(1):15-20. [CrossRef] [Medline]
    43. Benotsch EG, Kalichman S, Weinhardt LS. HIV-AIDS patients' evaluation of health information on the internet: the digital divide and vulnerability to fraudulent claims. J Consult Clin Psychol 2004 Dec;72(6):1004-1011. [CrossRef] [Medline]
    44. Bernhardt JM, Felter EM. Online pediatric information seeking among mothers of young children: results from a qualitative study using focus groups. J Med Internet Res 2004 Mar 1;6(1):e7 [FREE Full text] [CrossRef] [Medline]
    45. Birru MS, Monaco VM, Charles L, Drew H, Njie V, Bierria T, et al. Internet usage by low-literacy adults seeking health information: an observational analysis. J Med Internet Res 2004 Sep 3;6(3):e25 [FREE Full text] [CrossRef] [Medline]
    46. Borzekowski DL, Rickert VI. Adolescent cybersurfing for health information: a new resource that crosses barriers. Arch Pediatr Adolesc Med 2001 Jul;155(7):813-817. [Medline]
    47. Clayman ML, Manganello JA, Viswanath K, Hesse BW, Arora NK. Providing health messages to Hispanics/Latinos: understanding the importance of language, trust in health information sources, and media use. J Health Commun 2010;15 Suppl 3:252-263 [FREE Full text] [CrossRef] [Medline]
    48. Dart J, Gallois C, Yellowlees P. Community health information sources--a survey in three disparate communities. Aust Health Rev 2008 Feb;32(1):186-196. [Medline]
    49. Dutta-Bergman M. Trusted online sources of health information: differences in demographics, health beliefs, and health-information orientation. J Med Internet Res 2003;5(3):e21 [FREE Full text] [CrossRef] [Medline]
    50. Feufel MA, Stahl SF. What do web-use skill differences imply for online health information searches? J Med Internet Res 2012;14(3):e87 [FREE Full text] [CrossRef] [Medline]
    51. Gauld R, Williams S. Use of the Internet for health information: a study of Australians and New Zealanders. Inform Health Soc Care 2009 Sep;34(3):149-158. [CrossRef] [Medline]
    52. Ghaddar SF, Valerio MA, Garcia CM, Hansen L. Adolescent health literacy: the importance of credible sources for online health information. J Sch Health 2012 Jan;82(1):28-36. [CrossRef] [Medline]
    53. Helft PR, Eckles RE, Johnson-Calley CS, Daugherty CK. Use of the internet to obtain cancer information among cancer patients at an urban county hospital. J Clin Oncol 2005 Aug 1;23(22):4954-4962 [FREE Full text] [CrossRef] [Medline]
    54. Hesse BW, Nelson DE, Kreps GL, Croyle RT, Arora NK, Rimer BK, et al. Trust and sources of health information: the impact of the Internet and its implications for health care providers: findings from the first Health Information National Trends Survey. Arch Intern Med 2005;165(22):2618-2624. [CrossRef] [Medline]
    55. Ishikawa Y, Nishiuchi H, Hayashi H, Viswanath K. Socioeconomic status and health communication inequalities in Japan: a nationwide cross-sectional survey. PLoS One 2012;7(7):e40664 [FREE Full text] [CrossRef] [Medline]
    56. Kalichman SC, Cherry C, Cain D, Weinhardt LS, Benotsch E, Pope H, et al. Health information on the Internet and people living with HIV/AIDS: information evaluation and coping styles. Health Psychol 2006 Mar;25(2):205-210. [CrossRef] [Medline]
    57. Knapp C, Madden V, Wang H, Sloyer P, Shenkman E. Internet use and eHealth literacy of low-income parents whose children have special health care needs. J Med Internet Res 2011;13(3):e75 [FREE Full text] [CrossRef] [Medline]
    58. Knapp C, Madden V, Marcu M, Wang H, Curtis C, Sloyer P, et al. Information seeking behaviors of parents whose children have life-threatening illnesses. Pediatr Blood Cancer 2011 May;56(5):805-811. [CrossRef] [Medline]
    59. Lawson R, Forbes S, Williams J. Patterns of trust in sources of health information. N Z Med J 2011 Jan 21;124(1328):98-105. [Medline]
    60. Mackert M, Kahlor L, Tyler D, Gustafson J. Designing e-health interventions for low-health-literate culturally diverse parents: addressing the obesity epidemic. Telemed J E Health 2009 Sep;15(7):672-677 [FREE Full text] [CrossRef] [Medline]
    61. Maguire PA, Reay RE, Looi JCL, Cubis J, Byrne GJ, Raphael B. Neither the internist nor the Internet: use of and trust in health information sources by people with schizophrenia. Aust N Z J Psychiatry 2011 Jun;45(6):489-497. [CrossRef] [Medline]
    62. Marazienė D, Klumbienė J, Tomkevičiūtė J, Misevičienė I. Sources and reasons for seeking health information by Lithuanian adults. Medicina (Kaunas) 2012;48(7):371-378 [FREE Full text] [Medline]
    63. Marrie RA, Salter AR, Tyry T, Fox RJ, Cutter GR. Preferred sources of health information in persons with multiple sclerosis: degree of trust and information sought. J Med Internet Res 2013;15(4):e67 [FREE Full text] [CrossRef] [Medline]
    64. Murray E, Lo B, Pollack L, Donelan K, Catania J, White M, et al. The impact of health information on the internet on the physician-patient relationship: patient perceptions. Arch Intern Med 2003 Jul 28;163(14):1727-1734. [CrossRef] [Medline]
    65. Neter E, Brainin E. eHealth literacy: extending the digital divide to the realm of health information. J Med Internet Res 2012;14(1):e19 [FREE Full text] [CrossRef] [Medline]
    66. Nguyen GT, Bellamy SL. Cancer information seeking preferences and experiences: disparities between Asian Americans and Whites in the Health Information National Trends Survey (HINTS). J Health Commun 2006;11 Suppl 1:173-180. [CrossRef] [Medline]
    67. Nwagwu WE. The Internet as a source of reproductive health information among adolescent girls in an urban city in Nigeria. BMC Public Health 2007;7:354 [FREE Full text] [CrossRef] [Medline]
    68. Oh KM, Kreps GL, Jun J, Chong E, Ramsey L. Examining the health information-seeking behaviors of Korean Americans. J Health Commun 2012 Aug;17(7):779-801. [CrossRef] [Medline]
    69. Richter JG, Becker A, Koch T, Nixdorf M, Schacher B, Monser R, et al. Internet use in rheumatology outpatients in 2006: gender less important. Clin Exp Rheumatol 2009;27(1):15-21. [Medline]
    70. Smith D. Health care consumer's use and trust of health information sources. Journal of Communication in Healthcare 2011 Oct;4(3):200-210. [CrossRef]
    71. Miller LMS, Bell RA. Online health information seeking: the influence of age, information trustworthiness, and search challenges. J Aging Health 2012 Apr;24(3):525-541. [CrossRef] [Medline]
    72. van der Vaart R, van Deursen AJAM, Drossaert CHC, Taal E, van Dijk JAMG, van de Laar MAFJ. Does the eHealth Literacy Scale (eHEALS) measure what it intends to measure? Validation of a Dutch version of the eHEALS in two adult populations. J Med Internet Res 2011;13(4):e86 [FREE Full text] [CrossRef] [Medline]
    73. van Deursen AJAM, van Dijk JAGM. Internet skills performance tests: are people ready for eHealth? J Med Internet Res 2011;13(2):e35 [FREE Full text] [CrossRef] [Medline]
    74. Yan YY. Online health information seeking behavior in Hong Kong: an exploratory study. J Med Syst 2010 Apr;34(2):147-153. [Medline]
    75. Zhao X. Cancer information disparities between U.S.- and foreign-born populations. J Health Commun 2010;15 Suppl 3:5-21. [CrossRef] [Medline]
    76. Zoellner J, Connell C, Bounds W, Crook L, Yadrick K. Nutrition literacy status and preferred nutrition communication channels among adults in the Lower Mississippi Delta. Prev Chronic Dis 2009 Oct;6(4):A128 [FREE Full text] [Medline]
    77. Zulman DM, Kirch M, Zheng K, An LC. Trust in the internet as a health resource among older adults: analysis of data from a nationally representative survey. J Med Internet Res 2011;13(1):e19 [FREE Full text] [CrossRef] [Medline]
    78. Balshem H, Helfand M, Schünemann HJ, Oxman AD, Kunz R, Brozek J, et al. GRADE guidelines: 3. Rating the quality of evidence. J Clin Epidemiol 2011 Apr;64(4):401-406. [CrossRef] [Medline]
    79. Norman CD, Skinner HA. eHEALS: The eHealth Literacy Scale. J Med Internet Res 2006;8(4):e27 [FREE Full text] [CrossRef] [Medline]
    80. Black PC, Penson DF. Prostate Cancer on the Internet—Information or Misinformation? The Journal of Urology 2006 May;175(5):1836-1842. [CrossRef]
    81. Korfage IJ, van den Bergh RCN, Essink-Bot ML. Deciding on PSA-screening - Quality of current consumer information on the Internet. Eur J Cancer 2010 Nov;46(17):3073-3081. [CrossRef] [Medline]
    82. Dutta MJ, Bodie GD. Web searching for health: Theoretical foundations and connections to health related outcomes. Web Search: Springer Series in Information Science and Knowledge Management 2008;14(4):235-254. [CrossRef]
    83. van der Vaart R, Drossaert CHC, de HM, Taal E, van de Laar MAFJ. Measuring actual eHealth literacy among patients with rheumatic diseases: a qualitative analysis of problems encountered using Health 1.0 and Health 2.0 applications. J Med Internet Res 2013;15(2):e27 [FREE Full text] [CrossRef] [Medline]
    84. Martin LT, Ruder T, Escarce JJ, Ghosh-Dastidar B, Sherman D, Elliott M, et al. Developing predictive models of health literacy. J Gen Intern Med 2009 Nov;24(11):1211-1216 [FREE Full text] [CrossRef] [Medline]
    85. Carlson MDA, Morrison RS. Study design, precision, and validity in observational studies. J Palliat Med 2009 Jan;12(1):77-82 [FREE Full text] [CrossRef] [Medline]
    86. Car J, Lang B, Colledge A, Ung C, Majeed A. Interventions for enhancing consumers' online health literacy. Cochrane Database Syst Rev 2011(6):CD007092. [CrossRef] [Medline]


    Abbreviations

    CINAHL: Cumulative Index to Nursing and Allied Health Literature
    eHEALS: eHealth Literacy Scale
    HINTS: Health Information National Trends Survey
    MeSH: Medical Subject Headings
    S-TOFHLA: Short Test of Functional Health Literacy in Adults
    TOFHLA: Test of Functional Health Literacy in Adults


    Edited by G Eysenbach; submitted 11.11.14; peer-reviewed by S Ghaddar, D Castaneda; comments to author 19.02.15; revised version received 01.03.15; accepted 16.03.15; published 07.05.15

    ©Nicola Diviani, Bas van den Putte, Stefano Giani, Julia CM van Weert. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 07.05.2015.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.