This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Web-based technology has recently become an important source for sharing health information with patients after an acute cardiac event. Therefore, consideration of patients’ perceived electronic health (eHealth) literacy skills is crucial for improving the delivery of patient-centered health information.
The aim of this study was to translate and adapt the eHealth Literacy Scale (eHEALS) to conditions in Norway, and to determine its psychometric properties. More specifically, we set out to determine the reliability (internal consistency, test-retest) and construct validity (structural validity, hypotheses testing, and cross-cultural validity) of the eHEALS in self-report format administered to patients after percutaneous coronary intervention.
The original English version of the eHEALS was translated into Norwegian following a widely used cross-cultural adaptation process. Internal consistency was calculated using Cronbach α. The intraclass correlation coefficient (ICC) was used to assess the test-retest reliability. Confirmatory factor analysis (CFA) was performed for a priori-specified 1-, 2-, and 3-factor models. Demographic, health-related internet use, health literacy, and health status information was collected to examine correlations with eHEALS scores.
A total of 1695 patients after percutaneous coronary intervention were included in the validation analysis. The mean age was 66 years, and the majority of patients were men (1313, 77.46%). Cronbach α for the eHEALS was >.99. The corresponding Cronbach α for the 2-week retest was .94. The test-retest ICC for eHEALS was 0.605 (95% CI 0.419-0.743,
This study provides new information on the psychometric properties of the eHEALS for patients after percutaneous coronary intervention, suggesting a multidimensional rather than unidimensional construct. However, the study also indicated a redundancy of items, indicating the need for further validation studies.
ClinicalTrials.gov NCT03810612; https://clinicaltrials.gov/ct2/show/NCT03810612
Electronic health (eHealth) delivery provides an opportunity to redesign and improve health care services and health information using web-based technologies that can be accessed over the internet following diagnosis and discharge from hospital [
To date, there has been limited evidence on PROMs that are most appropriate for assessing eHealth literacy. Systematic reviews have reported that the eHealth literacy scale (eHEALS) was the only PROM used to measure eHealth literacy in more than one study [
The eHEALS has been adapted to different languages in Asia [
Therefore, in this current study, hypotheses were tested and evaluated against existing knowledge. For instance, a lower eHEALS score has been demonstrated among people with chronic illnesses [
Hypotheses regarding the relationship between eHEALSa scores and demographic information, health-related internet use, health literacy, and health status based on previous evidence.
Variables | Evidence (relationship with eHEALS)a | CONCARD-PCI hypothesis | Analysis | |
|
|
|
|
|
|
Age | Weak [ |
Weak to moderate relationship | Pearson correlation |
|
Gender | Weak [ |
Weak relationship | |
|
Education | Weak [ |
Weak relationship | ANOVAb |
|
Employment | Weak [ |
Weak relationship | ANOVA |
|
|
|
|
|
|
Used the internet to find information about health | Weak [ |
Moderate relationship | |
|
Patient’s interest in using the internet for health information in general (frequency of information-seeking) | Significant [ |
Moderate relationship | Spearman correlation |
|
|
|
|
|
|
Ability to find good health information | Moderate [ |
Moderate relationship | Pearson correlation |
|
Appraisal of health information | Positive [ |
Moderate relationship | Pearson correlation |
Health status based on RAND-12c (mental and physical component) | Weak [ |
Moderate relationship | Pearson correlation |
aeHEALS: Electronic health literacy scale.
bANOVA: analysis of variance.
cRAND-12: 12-item short-form health survey.
Thus, the aim of this study was to translate and adapt the eHEALS to conditions in Norway, and to determine its psychometric properties. More specifically, we set out to determine the reliability (internal consistency, test-retest) and construct validity (structural validity, hypotheses testing, and cross-cultural validity) of the eHEALS in a self-report format administered to patients after percutaneous coronary intervention.
This validation study used a cross-sectional design and was part of a larger prospective multicenter cohort study, CONCARDPCI, which seeks to identify bottlenecks and hurdles in the patient journey, and to suggest the optimal timing of services and alignment with preferences for patients with coronary artery disease undergoing percutaneous coronary intervention [
The study included 1695 patients at index admission for percutaneous coronary intervention at three large Norwegian university hospitals from June 12, 2017 through December 2018 (
Flow chart of the inclusion process. PCI: percutaneous coronary intervention; TAVI: transcatheter aortic valve implantation.
Inclusion criteria were patients undergoing percutaneous coronary intervention, ≥18 years of age, living at home at the time of inclusion, and having access to electronic equipment with internet access at the time of inclusion. Exclusion criteria were the inability to speak Norwegian or inability to fill out the self-reported questionnaire due to reduced capacity. To prevent a substantial difference in follow-up time or participants responding based on different assumptions, the patients who were likely to die within less than 1 year were excluded from the study. Institutionalized patients, who may be less likely to have follow up by a primary health care provider or to use the internet to find health information, were also excluded. Similarly, patients undergoing percutaneous coronary intervention without stent implantation and patients undergoing percutaneous coronary intervention related to transcatheter aortic valve implantation or MitralClip often have other indications for the examination or treatment including other follow-up routines and were therefore also excluded from this study.
Self-reports relating to eHealth literacy, health-related internet use, health literacy, health status, as well as demographic information and clinical data identified through the Norwegian Registry for Invasive Cardiology and patient medical records were obtained before discharge from hospital after percutaneous coronary intervention. The self-report was administered using a pencil and paper survey delivered with other PROMs used as part of the CONCARDPCI study. A random subgroup of 100 patients was approached for an eHEALS retest after 2 weeks, 74 (74.0%) of whom completed the retest.
A cross-cultural adaptation process was conducted to reach equivalence between the original source and the Norwegian target version of the eHEALS [
Step 1: Forward translation
Two forward translations of the English eHEALS were made by two bilingual translators for whom the target language (Norwegian) was their mother tongue.
The translators worked independently, and wrote a report (TL1 and TL2) that identified challenging phrases and described their rationale for final translation choices. An example of a difficult phrase to translate into Norwegian was “health resources.”
The two translations were compared and discrepancies were identified.
Step 2: Synthesis
The research team synthesized the reports (TL1 and TL2) into one consensus version (TL3) and described how they resolved discrepancies.
Step 3: Back translation
Two individuals who had a good understanding of English and also spoke Norwegian fluently independently translated TL3 back into English (TL4 and TL5). Neither of the translators who spoke English as their native language was aware of the original version of the eHEALS.
Step 4: Synthesis and back translation
The research team agreed on the modified Norwegian version of the eHEALS (TL6).
The research team discussed the timing of administration and meaning of certain words and sentences, and the Likert-type scale.
Step 5: Instrument pilot testing
The prefinal version (TL6) was discussed with patient representatives and piloted before being employed in the large-scale cohort study.
A cognitive interview was conducted to test the feasibility and understanding of the items. The patients were asked to read the questionnaire items as well as the instructions.
Step 6: Revised instrument
The researchers evaluated the adapted eHEALS questionnaire (TL6) and all necessary changes were made.
Patients who answered that they did not have access to electronic equipment with internet access in the pilot found it challenging to answer the eHEALS items. Therefore, the research team decided that the eHEALS items had low relevance and released these patients from answering the questionnaire.
Demographic information collected included age, gender, civil status, smoking status, education level (secondary school, trade school, high school, college/university less than 4 years, college/university 4 years or more), and employment status (working, retired, or other, including sick leave, disability pension, seeking employment). Clinical data included medical history (peripheral vascular disease, stroke, myocardial infarction, diabetes, previous percutaneous coronary intervention, previous coronary artery bypass grafting, previous other heart surgery) and indication for percutaneous coronary intervention (stable angina pectoris, unstable angina pectoris, nonST-segment elevation myocardial infarction [NSTEMI] or ST-segment elevation myocardial infarction [STEMI]).
The original English eHEALS comprises eight items and assesses patients’ own perception of their knowledge, comfort, and perceived skills at finding, evaluating, and applying eHealth information [
To assess patients’ health-related internet use, the following two supplementary items in the eHEALS were used: 1. How useful do you feel the internet is in helping you in making decisions about your health? 2. How important is it for you to be able to access health resources on the internet? [
The health literacy questions were selected from the Health Literacy Questionnaire (HLQ), which assesses nine separate domains of health literacy. In this study, two domains reflecting skills to use resources and critical evaluation were used: HLQ domain 5 (appraisal of health information, 5 items) and HLQ domain 8 (ability to find good information, 5 items). The first domain had a 4-point response option scale (strongly disagree to strongly agree) and the second domain had a 5-point response option scale (ranging from cannot achieve or always difficult to always easy). The items in the appraisal of the health information domain were used to calculate a total mean score ranging between 1 and 4, and the items in the ability to find good information domain were used to calculate a total mean score between 1 and 5. A low HLQ score indicates that the respondent has difficulties within the domain, and a high score indicates greater health literacy ability. In the event of more than two missing items, the domain score was regarded as missing [
The 12-item short-form survey RAND-12 [
The hypotheses testing (convergent validity, known-groups validity, and divergent validity) regarding the relationship between eHEALS scores and demographic information, health-related internet use, health literacy, and health status was formulated a priori. The hypotheses are based on evidence from previous studies on eHEALS as summarized in
We investigated the psychometric properties of the Norwegian version of the eHEALS by assessing the construct validity of three aspects: structural validity, hypotheses testing, and cross-cultural validity [
The reliability of the eHEALS was assessed by determining its internal consistency and test-retest reliability. Test-retest reliability was calculated by the intraclass correlation coefficient (ICC). Internal consistency reliability (ie, how well the items on a tool fit together) was calculated using Cronbach α, in which α>.70 was considered to be acceptable [
Confirmatory factor analyses (CFAs) were used to validate the extent to which the a priori hypotheses concerning dimensions of the eHEALS construct, based on theory and previous analyses, fit the actual data. CFA was used to explore the model fit of eHEALS as a 1-factor model as recommended by the original scale developer [
The convergent validity and divergent validity between the eHEALS and other constructs were assessed by computing Pearson correlation coefficients (
SPSS (IBM Corp. Released 2016, IBM SPSS Statistics for Windows, Version 24.0; Armonk, NY, USA) was used for summary statistics and correlations, and for conducting statistical comparisons. Mplus (Computer software, 1998-2018, version 7) developed by BO Muthén and LK Muthen, was used to perform CFAs.
The study was approved by the Norwegian Regional Committee for Ethics in Medical Research (REK 2015/57). All patients provided written informed consent, and confidentiality and the right to withdraw from the study were assured. The study conformed with the ethical principles outlined in the Declaration of Helsinki.
A total of 1695 patients consented to participate in the study (
Demographic and clinical characteristics of patients after percutaneous coronary intervention (N=1695).a
Characteristic | Value | Na | |
Age (years), mean (SD) | 66 (10) | 1695 | |
Gender (male), n (%) | 1313 (77.46) | 1695 | |
|
|
1529 | |
|
Married/Living with partner | 1173 (76.72) |
|
|
Living alone | 356 (23.28) |
|
|
|
1561 | |
|
Current smoker | 372 (23.83) |
|
|
Previous smoker (>1 month) | 713 (45.68) |
|
|
Never smoked | 476 (30.49) |
|
|
|
1522 | |
|
Secondary school | 331 (21.75) |
|
|
Trade school | 543 (35.68) |
|
|
High school | 156 (10.25) |
|
|
College/university (<4 years) | 269 (17.67) |
|
|
College/university (≥4 years) | 223 (14.65) |
|
|
|
1544 | |
|
Working | 559 (36.20) |
|
|
Retired | 771 (49.94) |
|
|
Other (sick leave, disability pension, seeking employment) | 214 (13.86) |
|
|
|
1685 | |
|
Peripheral vascular disease | 129 (7.66) |
|
|
Stroke | 72 (4.27) |
|
|
Myocardial infarction | 346 (20.53) |
|
|
Diabetes | 314 (18.63) |
|
|
Previous PCIb | 426 (25.28) | |
|
Previous CABGc | 180 (10.68) |
|
|
Previous other heart surgery | 19 (1.13) |
|
|
|
1695 | |
|
SAPd | 473 (27.91) |
|
|
UAPe | 266 (15.69) |
|
|
NSTEMIf | 522 (30.80) |
|
|
STEMIg | 346 (20.41) |
|
|
Other | 88 (5.19) |
|
Access to electronic equipment with internet access, n (%) | 1402 (93.66) | 1497 | |
Used the internet to find information about health, n (%) | 980 (66.08) | 1483 |
aNumber of observations for each characteristic may not total 1695 because of missing data.
bPCI: percutaneous coronary intervention.
cCABG: coronary artery bypass grafting.
dSAP: stable angina pectoris.
eUAP: unstable angina pectoris.
fNSTEMI: nonST-segment elevation myocardial infarction.
gSTEMI: ST-segment elevation myocardial infarction.
The mean eHEALS score was 25.66 (SD 6.23). The highest mean of the eHEALS items was 3.40 and the lowest mean was 2.92. Among all respondents, 80% were most likely to select one (41%) or two (39%) response options across all items, with 34%-51% responding “undecided” and 22%-47% responding “agree” (
Cronbach α for the eHEALS was >.99 (
Mean (SD) scores and Cronbach α values of the eHEALSa, HLQb, and RAND-12c of patients after percutaneous coronary intervention (N=1659).
Item | Mean (SD) | Cronbach α | ||
eHEALSa | 25.66 (6.23) | >.999 | ||
|
|
|
||
|
HLQ 5e | 2.43 (0.66) | .844 | |
|
HLQ 8d | 3.22 (0.73) | .875 | |
|
|
|
||
|
PCSf12 | 43.93 (10.88) | N/Ag | |
|
MCSh12 | 46.48 (11.14) | N/A |
aeHEALS: eHealth literacy scale.
bHLQ: health literacy questionnaire.
cHLQ 5:Appraisal of health information.
dHLQ 8: Ability to find good health information.
eRAND-12: 12-item short-form health survey.
fPCS: physical composite score.
gN/A: not applicable; since PCS12 and MCS12 of RAND-12 are not computed as means or sum scores, there is no Cronbach α.
hMCS: mental health composite score.
The strong standardized factor loadings for the 1-factor model, ranging from 0.79 to 0.93, indicated promising item properties. The Chi square test of model fit (
For the 2-factor model, standard factor loadings ranged from 0.80 to 0.93. Similar to the 1-factor model, this model suggested a good fit based on the SRMR, CFI, and TLI, and a poor fit for the RMSEA (
Standard factor loadings in the 3-factor model ranged from 0.84 to 0.97. Similar to the 1- and 2-factor models, the CFA supported a good fit for the three indices SRMR, CFI, and TLI, whereas the RMSEAs remained high (
Goodness-of-fit indices of the eHEALSa 1-, 2-, and 3-factor structure model.
Model | Chi square (df) |
RMSEAb (90% CI) | SRMRc | CFId | TLIe |
Model 1f | 1649.256 (20) | 0.247 (0.237-0.257) | 0.045 | 0.966 | 0.952 |
Model 2g | 1482.130 (19) | 0.240 (0.230-0.251) | 0.040 | 0.969 | 0.955 |
Model 3h | 510.925 (17) | 0.148 (0.137-0.159) | 0.019 | 0.990 | 0.983 |
aeHEALS: eHealth literacy scale.
bRMSEA: root mean square error of approximation.
cSRMR: standardized root mean square residual.
dCFI: comparative fit index.
eTLI: Tucker-Lewis index.
f1-factor model: Factor 1:1-8 [
g2-factor model: Factor 1: 1-5, 8; Factor 2: 6, 7 [
h3-factor model: Factor 1: 1, 2; Factor 2: 3-5; Factor 3: 6-8 [
Electronic health literacy scale (eHEALS) 3-factor model proposed by Sudbury-Riley et al [
Pearson correlation analysis showed a weak negative correlation between the eHEALS score and age. An independent-sample
As shown in
The between-groups analysis of variance also indicated a difference according to employment groups and eHEALS scores (
Association between electronic health literacy scale (eHEALS) scores, gender, and age. The scale was linearly transformed to a 0-100 scale. The scale was linearly converted to an 8-40 scale (scale from 8 to 40 computed as 8 + [scale from 0 to 100] × [40 – 8]/100). The eHEALS scale in the figure is: 0=8, 20=14.4, 40=20.8, 60=27.2, 80=33.6, 100=40.
An independent sample
Pearson correlation analysis revealed a moderate positive correlation between the eHEALS score and the HLQ scale for appraisal of health information and the HLQ scale for ability to find good information (
Pearson correlation analysis revealed a weak positive correlation between the eHEALS score and self-reported health assessed with RAND-12 (
Group statistics and correlations between eHEALSa score, patients’ demographic, and other instruments.
Variable | Statistic | ||
|
|
|
|
|
Age, Pearson correlation coefficient | –0.206 | <.001 |
|
Gender, 95% CI | –3.38-1.95 | .60 |
|
Education, |
21.085 | <.001 |
|
Employment, |
19.615 | <.001 |
|
|
||
|
Use of internet, 95% CI | –21.40 to –17.21 | <.001 |
|
eHEALS supp. 1c, Spearman correlation coefficient | 0.587 | <.001 |
|
eHEALS supp. 2d, Spearman correlation coefficient | 0.574 | <.001 |
|
|
|
|
|
HLQe 5f, Pearson correlation coefficient | 0.380 | <.001 |
|
HLQ 8g, Pearson correlation coefficient | 0.561 (<.001) | <.001 |
|
|
||
|
Mental component, Pearson correlation coefficient | 0.116 | <.001 |
|
Physical component, Pearson correlation coefficient | 0.112 | <.001 |
aeHEALS; eHealth literacy scale.
bANOVA: analysis of variance.
ceHEALS supp.1: How useful do you feel the internet is in helping you in making decisions about your health?
deHEALS supp.2: How important is it for you to be able to access health resources on the internet?
eHLQ: health literacy questionnaire.
fHLQ domain 5: appraisal of health information.
gHLQ domain 8: ability to find good information.
hRAND-12: 12-item short-form health survey.
To our knowledge, this is the first study to determine the psychometric properties of the eHEALS in patients after percutaneous coronary intervention. The Norwegian translation of the eHEALS appears to have acceptable construct validity. However, the high internal item consistency and the high RMSEA suggest that the fit of the data to the hypothesized models based on existing knowledge is not entirely adequate to fully capture the construct validity in this setting.
The structural validity was confirmed by three (SRMR, CFI, and TLI) out of four goodness-of-fit indices, indicating an adequate fit of the three hypothesized models. The RMSEA was lower in the 3-factor model [
A high proportion of patients were most likely to select the response “undecided” or “agree” across all items, suggesting that most of the patients either considered themselves as neutral (neither disagree or agree) or relatively confident about their knowledge, comfort, and perceived skills at finding, evaluating, and applying eHealth information. Although there is inconclusive evidence about the number of categories in a response scale and whether the neutral category has an impact on measurement quality [
The current study indicates adequate discriminant validity of the eHEALS, and most of the demographic information and other instruments confirmed the hypotheses defined a priori. As confirmed in previous psychometric studies within general adult European populations [
As the promotion of eHealth literacy takes place within a larger context, the original scale developer recommended involving other groups engaged in the literacy sectors in the work on validating the eHEALS [
The current study has several methodological strengths and limitations that should be addressed. The stringent linguistic, cultural, and measurement adaptation procedures are likely to have contributed to strengthening the conduct of the study. However, the Norwegian eHEALS showed mixed psychometric performance, which is likely due to the context of an acute coronary event. This indicates that hospitalization can affect the response to this type of PROM. Another key strength of the study is the large sample size, which allowed us to investigate the correlations between eHEALS scores, other PROMs, and subgroups. However, the analysis of the translated eHEALS was determined to be specific to patients who underwent percutaneous coronary intervention and cannot be generalized to other scenarios. There is therefore a need to determine the psychometric properties of the eHEALS in a more diverse population and in other settings to provide empirical evidence of the generalizability of the Norwegian eHEALS. Finally, the study only determined the administration of the eHEALS in self-reported written format (paper and pencil) in a hospital acute care setting. Further work should explore other modes of administration, including online administration developed for eHealth sources such as tablets, smartphones, and email.
This study provides new information on the psychometric properties of the eHEALS for patients after percutaneous coronary intervention, suggesting that the eHEALS is a multidimensional construct. Nevertheless, the RMSEA is not entirely adequate to fully capture the construct validity based on existing knowledge, and further factorial validation studies are needed. The internal item consistency was very high, indicating a redundancy of items. There is nonetheless a need for more research on the psychometric properties of the eHEALS. Moreover, use of the eHEALS in this study identified areas of eHealth literacy that are important for the further development of eHealth as a source of health information.
Response frequencies (%) and mean (SD) for all the eHEALS items, including supplementary items (N=1695).
confirmatory factor analysis
comparative fit index
Consensus-based standards for the selection of health status measurement instruments
electronic health literacy scale
electronic health
health literacy questionnaire
intraclass correlation coefficient
nonST-segment elevation myocardial infarction
patient reported outcome measure
12-item short-form health survey
root mean square error of approximation
standardized root mean square residual
ST-segment elevation myocardial infarction
Strengthening the Reporting of Observational Studies in Epidemiology
Tucker-Lewis index
The authors are grateful to Trond R Pettersen for forward translation, and Marie Norekvål Hayes and Alexander Sæløen for backward translation. Furthermore, Jon Martin Brørs and Rune Stiansen participated in the expert group. The authors are grateful to the CONCARDPCI team for including the patients, and to the patients who agreed to participate in the study and share their experiences. The authors thank Marie Norekvål Hayes for development of
None declared.