This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
The Internet increases the availability of health information, which consequently expands the amount of skills that health care consumers must have to obtain and evaluate health information. Norman and Skinner in 2006 developed an 8-item self-report eHealth literacy scale to measure these skills: the eHealth Literacy Scale (eHEALS). This instrument has been available only in English and there are no data on its validity.
The objective of our study was to assess the internal consistency and the construct and predictive validity of a Dutch translation of the eHEALS in two populations.
We examined the translated scale in a sample of patients with rheumatic diseases (n = 189; study 1) and in a stratified sample of the Dutch population (n = 88; study 2). We determined Cronbach alpha coefficients and analyzed the principal components. Convergent validity was determined by studying correlations with age, education, and current (health-related) Internet use. Furthermore, in study 2 we assessed the predictive validity of the instrument by comparing scores on the eHEALS with an actual performance test.
The internal consistency of the scale was sufficient: alpha = .93 in study 1 and alpha = .92 in study 2. In both studies the 8 items loaded on 1 single component (respectively 67% and 63% of variance). Correlations between eHEALS and age and education were not found. Significant, though weak, correlations were found between the eHEALS and quantity of Internet use (
The eHEALS was assessed as unidimensional in a principal component analysis and the internal consistency of the scale was high, which makes the reliability adequate. However, findings suggest that the validity of the eHEALS instrument requires further study, since the relationship with Internet use was weak and expected relationships with age, education, and actual performance were not significant. Further research to develop a self-report instrument with high correlations with people’s actual eHealth literacy skills is warranted.
Although a large supply of health information is available to educate and empower people, many lack the capability to use this information for their own benefit [
With the increased diffusion of the Internet among households, the accessibility to relevant health information for the public has increased spectacularly. Controversially, this might also further enlarge the existing differences in health knowledge and access to care [
Insight into people’s literacy skills is required to properly deploy guidelines, strategies, and interventions to offer information on different levels and in different formats. This is essential to make health information available and understandable to everyone who needs it [
To measure health literacy levels, the Rapid Estimate of Adult Literacy in Medicine (REALM) [
The eHEALS is an 8-item scale that tends to measure perceived skills at finding, evaluating, and applying electronic health information to health problems [
Two populations were studied, one containing patients with rheumatic diseases (study 1) and one containing a stratified sample of the general Dutch population (study 2). Because there are no other instruments that measure eHealth literacy, we measured convergent validity using the associated items age, education, and (health-related) Internet use. Predictive validity was measured by comparison with actual performance on various health-related Internet tasks [
A random sample of patients with rheumatic diseases was selected from the patient database of the rheumatology clinic of Medisch Spectrum Twente, Enschede, the Netherlands. A total of 496 patients were sent a personal invitation letter and a paper-and-pencil questionnaire by their treating rheumatologists. Patients expected to experience difficulty in completing the survey (e.g. because of significant cognitive impairment or illiteracy) were excluded a priori by their treating rheumatologists. The invitation letter explained the purpose of the study, the use of data, the voluntary nature, and the anonymity of the participant; therefore, returned questionnaires could be presumed to provide consent. A reminder was sent to those who did not respond within 2 weeks. According to local regulations in the Netherlands (Medical Research [Human Subjects] Act) the study did not need approval of the ethical review board; only (nonintervention) studies with a high burden for patients have to be reviewed. For this study, patients who indicated in the questionnaire that they did not have access to the Internet were excluded.
The questionnaire assessed the following: (1) gender, age, and education level, (2) general and health-related Internet use, and (3) the eHEALS. General Internet use was measured by 2 items: 1 yes/no item measuring access to the Internet, and 1 item on quantity of Internet use with answer options on a 5-point Likert scale ranging from “(almost) never” to “(almost) every day.” Health-related Internet use was measured with 8 items on quantity of use of different kinds of health-related information. Each item could be answered on a 4-point Likert scale ranging from “never” to “regularly” (see
A sample of 88 participants was recruited by randomly dialing telephone numbers in cities and villages in the region of Twente. A stratified sampling method was used to gain equal categories in gender, age, and education. When respondents indicated they were willing to participate, their contact and email address were recorded and a time for the research session was scheduled. All research sessions were scheduled at the University of Twente, which was an unfamiliar environment to all participants. Respondents received a follow-up letter in the mail for confirmation, and the day before the study respondents were reminded of the session by telephone. Respondents were awarded €25 for their participation.
The sessions lasted approximately 1.5 hours and started off with a short questionnaire that assessed (1) gender, year of birth, and education level, (2) general Internet use, and (3) the eHEALS. General Internet use was measured with 3 items: 1 yes/no item measuring access to the Internet, 1 item measuring amount of Internet use in hours per week, and 1 item on Internet experience in years.
Subsequently, participants had to complete a performance test, which contained nine health-related assignments, based on the four defined Internet skills. Two assignments (consisting of eight tasks) were used to measure operational Internet skills (e.g. open a health website, save a file, or add a website to the Favorites menu), two assignments (consisting four tasks) were used to measure formal Internet skills (e.g. navigate different health-related menu and website designs, and surf between different websites), three assignments were used to measure information Internet skills (find health-related information on the Internet), and two assignments were used to measure strategic Internet skills (e.g. extract information from different sources, and make decisions based on the information found). The assignments were generated by a team of researchers that made a conscious effort to include only tasks that were accessible and relevant to the general user population (e.g. find the Web address of a health clinic, or search for information on vitamins). All assignments were pilot tested with 12 participants to ensure comprehensibility and applicability. Assignments were administered in a sequence of increasing difficulty, as indicated in
Data were analyzed using SPSS version 17.0 for Windows (IBM Corporation, Somers, NY, USA) in both studies. Cronbach alpha served as a measure of internal consistency, reflecting the (weighted) average correlation of items within the scale [
Distributional properties of the eHEALS were further inspected to examine the normality of the total scores and to identify floor and ceiling effects. Skewness and kurtosis values between ±1 were assumed to indicate no or slight nonnormality. Floor or ceiling effects were considered to be present if >15% of the participants scored the worst or the best possible score on the eHEALS [
Evidence for convergent validity was determined by studying Spearman correlations between total mean scores on the eHEALS and age, education level, quantity of Internet use, and sum scores of health-related Internet use. Based on previous studies on regular health literacy, we hypothesized negative correlations with age and positive correlations with education and (health-related) Internet use [
Of the 496 invitations sent out, 12 were returned undeliverable. In total, 227 of 484 questionnaires were returned (47%); 189 of these 227 participants had Internet access and completed the eHEALS (83%). Participant characteristics and Internet use are shown in
Participants’ self-reported sociodemographics and (health-related) Internet use
Study 1 (n = 189) n (%) | Study 2 (n = 88) n (%) | ||
|
|||
Male | 119 (63) | 45 (51) | |
Female | 70 (37) | 43 (49) | |
Mean (SD) age (years) | 52 (11) | 43 (18) | |
|
|||
Low | 38 (20) | 25 (28) | |
Middle | 102 (54) | 32 (36) | |
High | 46 (24) | 31 (35) | |
Unknown | 3 (2) | ||
|
|||
(almost) Every day | 117 (62) | –a | |
Several days a week | 34 (18) | ||
About 1 day a week | 15 (8) | ||
Less than 1 day a week | 9 (5) | ||
(almost) Never | 12 (6) | ||
Unknown | 2 (1) | ||
Mean (SD) Amount of Internet use (hours per week) | –a | 12.2 (13.7) | |
Mean (SD) Internet experience (years) | –a | 9.3 (4.3) | |
|
|||
Diseases | 159 (84) | –a | |
Healthy lifestyle | 121 (64) | ||
Medication | 95 (50) | ||
Treatments | 122 (65) | ||
Care providers | 69 (37) | ||
Patient organizations | 67 (35) | ||
Law regulations related to health conditions | 61 (34) | ||
Peer-support forums | 45 (24) |
a Item was not measured in this study.
Total scores on the eHEALS were approximately normally distributed with a skewness of -.63. Floor and ceiling effects were acceptable, with no participants scoring the worst possible score (8), and 5 participants scoring the best possible score (40).
The internal consistency of the eHEALS was alpha = .93. Unidimensionality of the scale was supported by principal component analysis (eigenvalue = 5.4, 67% of variance explained). The eigenvalue of the first component was 5 times larger than the eigenvalue of the second component (being 1.1). All items loaded high on this component, ranging from .74 to .85 (
eHealth Literacy Scale (eHEALS) mean items scores, scale reliability, and principal component analysis
Item | Study 1 | Study 2 | Factor loading | item-total correlationa | ||||
Mean | SD | Mean | SD | Study 1 | Study 2 | Study 1 | Study 2 | |
1: I know what health resources are available on the Internet | 3.6 | 0.83 | 3.4 | 0.86 | .82 | .77 | .80 | .70 |
2: I know where to find helpful health resources on the Internet | 3.6 | 0.87 | 3.3 | 0.88 | .85 | .79 | .84 | .73 |
3: I know how to find helpful health resources on the Internet | 3.7 | 0.81 | 3.5 | 0.94 | .85 | .86 | .85 | .72 |
4: I know how to use the Internet to answer my health questions | 3.6 | 0.85 | 3.6 | 0.88 | .83 | .86 | .83 | .70 |
5: I know how to use the health information I find on the Internet to help me | 3.5 | 0.88 | 3.4 | .087 | .84 | .77 | .85 | .67 |
6: I have the skills I need to evaluate the health resources I find on the Internet | 3.6 | 0.89 | 3.6 | 0.90 | .82 | .77 | .84 | .67 |
7: I can tell high-quality from low-quality health resources on the Internet | 3.4 | 0.95 | 3.4 | 1.00 | .80 | .75 | .82 | .76 |
8: I feel confident in using information from the Internet to make health decisions | 3.3 | 0.99 | 3.1 | 1.12 | .74 | .80 | .78 | .82 |
Mean (SD) sum score | 28.2 | 5.9 | 27.6 | 5.9 | ||||
Eigenvalue first component | 5.36 | 5.06 | ||||||
Variance accounted for | 67% | 63% | ||||||
Cronbach alpha | .93 | .92 |
a All item-total correlations were significant at
Spearman correlations between scores on the eHealth Literacy Scale (eHEALS) and age, education, (health-related) Internet use, and Internet performance skills
Study 1 | Study 2 | ||||
|
|
|
|
||
|
|||||
Age | -.11 | .13 | -.08 | .49 | |
Education (1 = low, 2 = middle, 3 = high) | .09 | .24 | .13 | .25 | |
Amount of Internet usage | .24 | .001 | .24 | .02 | |
|
|||||
Information on diseases | .40 | <.001 | –a | ||
Healthy lifestyle | .28 | <.001 | |||
Medication | .29 | <.001 | |||
Treatments | .38 | <.001 | |||
Care providers | .30 | <.001 | |||
Patient organizations | .32 | <.001 | |||
Law regulations related to health conditions | .26 | <.001 | |||
Peer-support forums | .27 | <.001 | |||
|
|||||
Successfully completed tasks overall | –a | .18 | .09 | ||
Operational | .12 | .27 | |||
Formal | .19 | .07 | |||
Information | .05 | .62 | |||
Strategic | .11 | .30 |
a Item was not measured in this study.
Characteristics and Internet use of the 88 recruited participants in study 2 are shown in
Overview of proportion of tasks successfully completed in performance tests
Internet skills (number of tasks) | Average task completion | ||
Mean | SD | % | |
Operational tasks (8) | 5.8 | 2.1 | 73 |
Formal tasks (4) | 2.9 | 1.2 | 73 |
Information tasks (3) | 1.5 | 0.9 | 50 |
Strategic tasks (2) | 0.7 | 0.8 | 35 |
As in study 1, total scores on the eHEALS were approximately normally distributed with a slight skewness of –.80. Floor and ceiling effects were acceptable, with no participants scoring the worst possible score (8), and 4 participants scoring the best possible score (40).
The internal consistency of the eHEALS was alpha = .92. All items loaded on 1 single component in this study as well (eigenvalue = 5.1, 63% of variance explained). The eigenvalue of the first component was 5.8 times larger than the eigenvalue of the second component (being .88). All items loaded high on this component, ranging from .75 to .86 (
No significant correlations between the eHEALS and either age (
eHealth Literacy Scale (eHEALS) mean scores of participants scoring below and above median scores on performance tasks
Performance tasks | Mean | SD |
|
df |
|
|
|
||||||
50% below median | 3.38 | 0.85 | –.998 | 80.33 | .32 | |
50% above median | 3.53 | 0.59 | ||||
|
||||||
50% below median | 3.36 | 0.77 | –1.47 | 77.38 | .15 | |
50% above median | 3.59 | 0.67 | ||||
|
||||||
50% below median | 3.43 | 0.69 | –.26 | 81.37 | .80 | |
50% above median | 3.47 | 0.80 | ||||
|
||||||
50% below median | 3.38 | 0.74 | –.79 | 81.55 | .43 | |
50% above median | 3.51 | 0.74 |
The results of the two studies show that the eHEALS is unidimensional and has high internal consistency. Yet results of the validity tests showed that the eHEALS is not a valid measure of eHealth literacy.
With regard to the convergent validity, we hypothesized at least moderate positive correlations (
We hypothesized at least moderately positive correlations (
Concerning the predictive validity, the lack of significant correlations between the eHEALS and actual performance skills was surprising. Since the assignments used in study 2 were applicable to the general Internet user, one would at least expect some moderate correlations between the eHEALS scale and the performance results. Apparently, perceived skills (as obtained with eHEALS) do not predict actual performance (as measured in study 2). Previous investigations on general computer skills have also shown that people tend to overestimate their computer skills, which results in a gap between self-reported skills and practice when actual skills are measured [
We suggest a revision of the eHEALS, in a way that all four different skills are measured: (1) operational and (2) formal skills that measure practical use of computers and the Internet, and (3) information finding and (4) strategic skills that measure search strategies and skills to judge the found information. Also, questions might need to be formulated differently in order to prevent misunderstanding or differing interpretations. To this aim, qualitative research might provide more insight into the basis for participants’ answers—for example, having people fill out the eHEALS with techniques such as cognitive interviewing or thinking-aloud methods [
A limitation of both our studies is the voluntary basis on which participants were recruited. This could have caused a bias, because participants might already have been more interested in using the Internet and searching for information, which could have influenced the results. Concerning study 1, only patients with rheumatic diseases were invited to participate. Therefore, this study might not be representative for other chronic conditions, since patients with rheumatic diseases are on average somewhat older. Concerning study 2, because of the major labor intensity of performance tests and the very high travel costs of bringing participants nationwide to the university lab, it was not possible to test a random sample from the whole Dutch population. Although the study population size of 88 is not enough to generalize to the whole population, the applied quota sample for the categories of gender, age, and education hugely improved representativeness.
The eHEALS is found to be unidimensional, according to principal component analysis, and to be internally consistent, as assessed with Cronbach alpha, but its validity is questionable. Expected correlations between the eHEALS and peoples’ use of the Internet were weak. Moreover, scores on the eHEALS did not correlate with age, education, and scores on performance tasks, and the eHEALS was not able to distinguish between people with high and low health-related Internet skills. Therefore, more research is needed in order to develop a self-report instrument that validly measures eHealth literacy skills. We suggest incorporation of operational, formal, information, and strategic Internet skills to measure all aspects of eHealth literacy.
The work of study 1 was supported by an unrestricted educational grant from Wyeth Pharmaceuticals, part of Pfizer Inc. This funding source had no involvement in data collection, analysis, or the preparation of this manuscript.
None declared
Performance test assignments.