This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.
Digital technologies have changed how we manage our health, and eHealth literacy is needed to engage with health technologies. Any eHealth strategy would be ineffective if users’ eHealth literacy needs are not addressed. A robust measure of eHealth literacy is essential for understanding these needs. On the basis of the eHealth Literacy Framework, which identified 7 dimensions of eHealth literacy, the eHealth Literacy Questionnaire (eHLQ) was developed. The tool has demonstrated robust psychometric properties in the Danish setting, but validity testing should be an ongoing and accumulative process.
This study aims to evaluate validity evidence based on test content, response process, and internal structure of the eHLQ in the Australian community health setting.
A mixed methods approach was used with cognitive interviewing conducted to examine evidence on test content and response process, whereas a cross-sectional survey was undertaken for evidence on internal structure. Data were collected at 3 diverse community health sites in Victoria, Australia. Psychometric testing included both the classical test theory and item response theory approaches. Methods included Bayesian structural equation modeling for confirmatory factor analysis, internal consistency and test-retest for reliability, and the Bayesian multiple-indicators, multiple-causes model for testing of differential item functioning.
Cognitive interviewing identified only 1 confusing term, which was clarified. All items were easy to read and understood as intended. A total of 525 questionnaires were included for psychometric analysis. All scales were homogenous with composite scale reliability ranging from 0.73 to 0.90. The intraclass correlation coefficient for test-retest reliability for the 7 scales ranged from 0.72 to 0.95. A 7-factor Bayesian structural equation modeling using small variance priors for cross-loadings and residual covariances was fitted to the data, and the model of interest produced a satisfactory fit (posterior productive
The evidence suggests that the eHLQ is a tool with robust psychometric properties and further investigation of discriminant validity is recommended. It is ready to be used to identify eHealth literacy strengths and challenges and assist the development of digital health interventions to ensure that people with limited digital access and skills are not left behind.
Digital technologies have brought fundamental changes to modern-day life including how we manage our health. We can quickly search for health information at our fingertips but are also facing an avalanche of misinformation, as evident during the COVID-19 pandemic [
To characterize the challenges of accessing and using digital technologies for health, the concept of eHealth literacy was coined in 2006 [
To describe and address eHealth literacy needs, Norgaard et al [
Using technology to process health information
Understanding of health concepts and language
Ability to actively engage with digital services
Feel safe and in control
Motivated to engage with digital services
Access to digital services that work
Digital services that suit individual needs [
With the inclusion of eHealth literacy dimensions relating to user interaction and experiences in using digital health systems, the eHLQ embraces the real-world experiences of users while capturing the interactivity and increasing capabilities of digital technologies. It can provide rich information about the competencies of individuals as well as the maturity of digital health systems, as mature systems are likely to be more responsive to the individual needs of users [
The eHLQ was simultaneously developed in Danish and English to avoid idiom and improve item wording to enhance future translation of the questionnaire into other languages [
Validity testing is an ongoing process that involves the accumulation of 5 sources of evidence based on test content, response process, internal structure, relations to other variables, and consequences of testing, according to the authoritative reference of developing and using of educational and psychological measurements, the
Evidence based on test content is used to determine whether the items represent the content domain and may also include whether the wordings are easy to read and formats of administration are easy to use. Response process refers to the cognitive process of survey participants, that is, whether the interpretation of the items by participants aligns with the intended interpretation of items by test developers. It may also include whether interpretation remains the same across subgroups or across different formats of administration. Internal structure is the extent to which items conform to the constructs and relates to aspects such as factor analysis, reliability, and measurement invariance. Relations to other variables is the analysis of the relationship between the scores on another instrument relevant in the theoretical network of the construct being measured or other external variables that the scores can predict, whereas consequences of testing relates to the robustness of the proposed use of the test scores, including intended benefits, indirect effects, and unintended consequences such as construct underrepresentation or construct irrelevance [
Methods to collect and evaluate the validity evidence were guided by the discussion in the
Eligibility criteria for participation in both activities were clients aged ≥18 years, with or without any health conditions, and able to complete the questionnaire in paper-based format, web-based format, or face-to-face interview. Clients experiencing significant cognitive or mental health issues, who were too clinically unwell, and with insufficient English to complete the questionnaire and who did not have a carer to assist them were excluded.
Ethical approval of the study was obtained from the Deakin University Human Research Ethics Committee (HEAG-H 146_2017).
Cognitive interviewing is commonly used to explore the cognitive process of how people answer survey items [
Given the qualitative nature of cognitive interviewing, a large sample size is not required but needs to be representative and diverse [
Participants were recruited with assistance from the health site, and a plain language information sheet was provided, with written consent requested. Interviews were conducted after participants completed the paper-based format. Participant behavior was observed when they answered the questionnaire. Upon completion, two questions were asked to gain insights into the cognitive process: (1) What were you thinking when you answered this question? (2) Why did you choose this answer? Participants were encouraged to make any further comments about the items or the format. They could be interviewed for all 35 items or part of the questionnaire, mainly for older participants to avoid fatigue and cognitive overload.
Data analysis was conducted using text summary [
For the cross-sectional survey, clients were recruited at the waiting area and provided with an information sheet. A signed consent form was not requested with the return of the completed questionnaire as implied consent as a strategy to facilitate participation. Apart from self-administration using paper-based or web-based formats, interviews were included to ensure that older people or people with low literacy could participate. Demographic questions including age, sex, postcode, language spoken at home, education, health status, perceived health status, and use of digital services were also collected. Further description of recruitment is described in the study by Cheng et al [
Similar to the Danish eHLQ validity testing [
Analyses were conducted using three statistical software programs, namely, SPSS (version 25.0; IBM Corp) [
To deal with missing data for the eHLQ scores, the data set was first examined. If no clear pattern of missingness was found, that is, the missingness could be regarded as completely at random, a 2-step approach would be taken. The first step was to delete cases with more than 50% of missing values to reduce potential bias. The second step was to replace all missing values using the expectation-maximization algorithm imputation in SPSS [
Item difficulty is an item property and is usually conducted as a first step in item analysis in the CTT approach [
To measure reliability, internal consistency and test-retest reliability were evaluated. In addition to the commonly used Cronbach
Following the classical item analysis, CFA was conducted, given that the hypothesized constructs were specified a priori. Similar to the Danish validity testing, the BSEM approach was adopted [
The different parameter specifications in BSEM at the start of an analysis are described as priors, which can be diffuse (noninformative) or informative [
A sequence of 1-factor models followed by a 7-factor model (
On the basis of the results of the selected 7-factor model, discriminant validity was evaluated using the Fornell-Larcker criteria [
The BSEM approach is further used to test for differential item functioning (DIF), that is, the stability of measurement across different groups or grouping variables [
Bayesian multiple-indicators, multiple-causes model for differential item functioning testing with scale 1 of eHealth Literacy Questionnaire as the example. Output from Mplus: Admin: administration format (0=face-to-face interview, 1=paper format); Area: site area, that is, location of participating organization (0=metropolitan, 1=regional); Setting: health setting (0=private clinic, 1=community health); UTPHI: eHealth Literacy Questionnaire scale 1 (using technology to process health information); Q7D1, Q11D1, Q13D1, Q20D1, and Q25D1: eHealth Literacy Questionnaire items.
To perform an IRT analysis, 4 assumptions need to be met. The assumptions of unidimensionality (items are measuring the same construct), local independence (each item should not be related except they are measuring the same construct), and item invariance (item parameters are the same across subgroups) can be examined through the CTT methods described in the CTT Analysis section. The assumption of monotonicity (the probability of endorsing an item increases as the trait level increases) is evaluated by examining the test characteristic curves [
A total of 12 participants were recruited for 2 rounds of cognitive interviews. Of these 12 participants, 8 (67%) were women and 4 (33%) were men, with 58% (7/12) of the participants aged >65 years and 17% (2/12) speaking a language other than English at home. The sample provided a good representation of people from a range of different age groups and cultural backgrounds. The first round with 7 participants identified the term
Despite the diverse backgrounds of participants, no major differences in understanding the items were identified, and all participants found the items easy to read. Recommendations from participants also led to changes in the introductory page to provide examples of technology, health technology, eHealth systems, and health care providers or health professionals. The completion time of the questionnaire ranged from 3 to <7 minutes.
A total of 530 completed questionnaires were collected. On the basis of the treatment of missing values described in the Statistical Analysis section, 5 cases were deleted, leading to a final sample size of 525 for psychometric analyses. The age of participants of the cross-sectional survey ranged from 18 to 94 years, and 61% (320/525) of the participants were women. A total of 33.3% (175/525) of the participants had a university education, and 30.9% (162/535) spoke a language other than English at home. Of the 525 participants, 300 (57.1%) reported having some form of chronic illness. Regarding technology use, of the 525 participants, 151 (28.8%) did not have a computer or laptop, and 131 (25%) did not use email or SMS text messaging (
Characteristics of cross-sectional survey participants (N=525).
Characteristics | Value | ||
Age (years), mean (SD; range) | 56.8 (18.6; 18-94) | ||
|
|||
|
Site 1: metropolitan private clinic | 204 (38.9) | |
|
Site 2: metropolitan community health | 204 (38.9) | |
|
Site 3: regional private clinic | 117 (22.3) | |
|
|||
|
Paper-based | 399 (76) | |
|
Web-based | 13 (2.5) | |
|
Face-to-face interview | 113 (21.5) | |
|
|||
|
Female | 320 (61) | |
|
Male | 203 (38.7) | |
|
|||
|
Primary school or below | 27 (5.1) | |
|
Secondary school or below | 173 (33) | |
|
Trade certificate or diploma | 141 (26.9) | |
|
Completed university | 175 (33.3) | |
|
|||
|
English | 363 (69.1) | |
|
Other | 161 (30.7) | |
|
IRSDb 1-4 | 123 (23.5) | |
|
IRSD 5-6 | 111 (21.1) | |
|
IRSD 7-8 | 134 (25.5) | |
|
IRSD 9-10 | 140 (26.6) | |
|
|||
|
Yes | 249 (47.4) | |
|
No | 267 (50.9) | |
|
|||
|
No | 225 (42.9) | |
|
Arthritis | 115 (21.9) | |
|
Cancer | 14 (2.7) | |
|
Heart disease | 90 (17.1) | |
|
Diabetes | 67 (12.8) | |
|
Respiratory condition | 41 (7.8) | |
|
Anxiety | 69 (13.1) | |
|
Depression | 69 (13.1) | |
|
Other | 89 (17) | |
|
|||
|
Good to excellent | 400 (76.1) | |
|
Fair to poor | 103 (21.5) | |
|
|||
|
Computer or laptop | 374 (71.2) | |
|
Mobile phone or smartphone | 459 (87.4) | |
|
Tablet | 241 (45.9) | |
Number of devices owned, mean (SD; range) | 2.1 (0.9; 0-4) | ||
|
|||
|
394 (75) | ||
|
SMS text messaging | 398 (75.8) | |
|
266 (50.7) | ||
|
30 (5.7) | ||
|
104 (19.8) | ||
|
Snapchat | 51 (9.7) | |
|
WhatsApp or WeChat | 112 (21.3) | |
|
Blogging | 15 (2.9) | |
|
Forum/chat room | 26 (5) | |
Number of platforms used, mean (SD; range) | 2.7 (1.8; 0-10) | ||
Looked for web-based information in the last 3 months, n (%) | 392 (74.4) | ||
Monitored health digitally, n (%) | 183 (34.9) |
aSocioeconomic status is classified by IRSD10. This index is based on information provided by the Australian Bureau of Statistics [
bIRSD: Index of Relative Socio-economic Disadvantage Decile 2016 of Australia.
The mean scale scores ranged from 2.43 (SD 0.57) for scale 7 (
Observation during cognitive interviewing and the main survey did not identify any issue when people responded to the items for either the paper-based or web-based format. An inspection of the comments marked on the 530 completed questionnaires from the main survey found that 0.03% (15/530) of the participants put a question mark next to some items, indicating that they did not understand those items, while 0.10% (55/530) of the participants provided unclear answers. These results suggested that the items were generally understood, and the 4-point ordinal scale was acceptable.
eHealth Literacy Questionnaire scale scores (N=525; score range 1-4).
Scale | Value, mean (SD) | Missing data |
1. Using technology to process health information | 2.59 (0.61) | 0 |
2. Understanding of health concepts and language | 2.95 (0.41) | 0 |
3. Ability to actively engage with digital services | 2.65 (0.68) | 1 |
4. Feel safe and in control | 2.83 (0.49) | 5 |
5. Motivated to engage with digital services | 2.63 (0.55) | 0 |
6. Access to digital services that work | 2.64 (0.45) | 1 |
7. Digital services that suit individual needs | 2.43 (0.57) | 11 |
A range of item difficulty was found for all scales, reflecting a spectrum of difficulty levels across the relevant constructs. The scale with the smallest range of item difficulty was 7 (
The chosen 1-factor Bayesian models (with informative priors for residual covariances of
A subsequent 7-factor model was fitted to the data set with 6 models tested. All models fitted the data well. The model with priors for the variance of cross-loadings set to 0.01 and inverse-Wishart
Inspection of the AVE showed that the estimates of 4 scales met the first Fornell-Larcker criterion, whereas 3 scales were <0.50 (scales 2, 4, and 6). Given that these AVE estimates were based on the 7-factor model that allowed for cross-loadings and residual covariances, AVE estimates from the 1-factor models were also calculated, and the AVE estimates for the 7 scales were 0.66, 0.49, 0.72, 0.61, 0.65, 0.47, and 0.74. Hence, the AVE estimates of scales 2 and 6 were still <0.50. The second criterion of the factor’s association with other factors was also not satisfied. On the basis of this criterion, only scale 2 demonstrated good discrimination with scales 4, 6, and 7, and scale 4 had good discrimination with all scales except scale 6. Hence, there might not be sufficient discriminant validity among the scales (
For internal consistency, the Cronbach
The Bayesian MIMIC models for testing DIF for administration format, site area, and health setting achieved a good model fit. The model with a DIF path of 0.01 was chosen as the model of interest for the 7 scales with
For group differences, no significant differences were found for site area and health setting, but group differences were identified for the administration format with the self-administered paper-based format scoring higher than face-to-face interviews for scales 1 (
Factor loadings of the eHealth Literacy Questionnaire 7-factor Bayesian confirmatory factor analysis model with priors for cross-loadings of 0.01 and residual covariances of 150a.
Itemb | 1. Using technology | 2. Health concepts | 3. Ability | 4. Feel safe | 5. Motivated | 6. Access | 7. Suit needs | |
|
||||||||
|
Q7D1 |
|
0.02 | 0.02 | 0.00 | −0.08 | −0.06 | −0.06 |
|
Q11D1 |
|
0.03 | −0.01 | −0.01 | −0.02 | −0.04 | −0.08 |
|
Q13D1 |
|
−0.02 | −0.03 | 0.02 | 0.08 | 0.06 | 0.05 |
|
Q20D1 |
|
−0.03 | −0.01 | 0.02 | 0.05 | 0.06 | 0.08 |
|
Q25D1 |
|
−0.01 | 0.02 | 0.01 | 0.03 | 0.04 | 0.06 |
|
||||||||
|
Q5D2 | 0.06 |
|
0.03 | 0.00 | 0.04 | 0.01 | 0.01 |
|
Q12D2 | 0.02 |
|
0.01 | 0.02 | −0.02 | −0.03 | −0.03 |
|
Q15D2 | −0.04 |
|
−0.02 | 0.03 | −0.01 | 0.03 | 0.02 |
|
Q21D2 | −0.03 |
|
−0.01 | −0.01 | −0.03 | −0.02 | −0.02 |
|
Q26D2 | 0.02 |
|
−0.00 | −0.02 | 0.05 | 0.04 | 0.04 |
|
||||||||
|
Q4D3 | 0.00 | −0.00 |
|
0.04 | 0.03 | 0.03 | 0.03 |
|
Q6D3 | 0.02 | 0.01 |
|
0.03 | −0.02 | −0.04 | −0.05 |
|
Q8D3 | 0.03 | 0.02 |
|
0.01 | 0.03 | 0.02 | 0.03 |
|
Q17D3 | 0.00 | −0.01 |
|
−0.02 | −0.03 | −0.03 | −0.04 |
|
Q32D3 | −0.03 | 0.01 |
|
−0.04 | 0.01 | 0.03 | 0.07 |
|
||||||||
|
Q1D4 | 0.02 | 0.00 | 0.01 |
|
−0.01 | −0.02 | −0.03 |
|
Q10D4 | 0.05 | 0.02 | 0.01 |
|
0.04 | 0.01 | 0.00 |
|
Q14D4 | 0.04 | 0.05 | 0.02 |
|
0.05 | 0.05 | 0.03 |
|
Q22D4 | −0.03 | −0.02 | −0.01 |
|
−0.03 | −0.01 | 0.00 |
|
Q30D4 | −0.01 | −0.01 | 0.01 |
|
0.01 | 0.02 | 0.04 |
|
||||||||
|
Q2D5 | −0.04 | −0.02 | −0.01 | 0.02 |
|
−0.01 | −0.02 |
|
Q19D5 | 0.04 | 0.02 | 0.02 | −0.04 |
|
−0.00 | 0.00 |
|
Q24D5 | −0.02 | 0.01 | −0.02 | 0.05 |
|
0.02 | 0.01 |
|
Q27D5 | 0.00 | 0.01 | −0.01 | 0.01 |
|
−0.01 | −0.02 |
|
Q35D5 | 0.04 | 0.00 | 0.03 | −0.01 |
|
−0.00 | 0.00 |
|
||||||||
|
Q3D6 | -0.11 | 0.02 | −0.05 | 0.06 | −0.08 |
|
−0.08 |
|
Q9D6 | 0.13 | −0.00 | 0.08 | −0.03 | 0.05 |
|
0.05 |
|
Q16D6 | −0.11 | 0.02 | −0.04 | 0.05 | −0.05 |
|
−0.02 |
|
Q23D6 | 0.05 | −0.03 | 0.01 | 0.00 | 0.03 |
|
0.01 |
|
Q29D6 | 0.00 | −0.01 | −0.01 | −0.01 | 0.02 |
|
−0.00 |
|
Q34D6 | 0.12 | 0.01 | 0.08 | −0.07 | 0.07 |
|
0.07 |
|
||||||||
|
Q18D7 | 0.05 | 0.02 | 0.04 | −0.02 | −0.02 | −0.03 |
|
|
Q28D7 | 0.00 | −0.03 | −0.02 | 0.01 | 0.03 | 0.01 |
|
|
Q31D7 | −0.09 | 0.01 | −0.05 | 0.07 | −0.04 | 0.03 |
|
|
Q33D7 | 0.02 | 0.00 | 0.04 | −0.04 | −0.00 | −0.02 |
|
aModel fit: posterior predictive
bSee
cItalicized values indicate statistically significant factor loadings (
Interfactor correlations (below diagonal), average variance extracted (diagonal), and shared variance estimates (above diagonal) for the 7 eHealth Literacy Questionnaire scales.
Scale | 1. Use tech | 2. Health concepts | 3. Ability | 4. Feel safe | 5. Motivated | 6. Access | 7. Suit needs |
1. Using technology to process health information | 0.53a | 0.37b | 0.90b | 0.06 | 0.84b | 0.38b | 0.56b |
2. Understanding of health concepts and language | 0.61 | 0.32a | 0.38b | 0.22 | 0.34b | 0.25 | 0.21 |
3. Ability to actively engage with digital services | 0.95 | 0.62 | 0.59a | 0.04 | 0.72b | 0.34 b | 0.61b |
4. Feel safe and in control | 0.25 | 0.47 | 0.21c | 0.47a | 0.12 | 0.34b | 0.19 |
5. Motivated to engage with digital services | 0.91 | 0.58 | 0.85 | 0.35 | 0.52a | 0.63b | 0.69b |
6. Access to digital services that work | 0.62 | 0.50 | 0.58 | 0.58 | 0.80 | 0.32a | 0.75b |
7. Digital services that suit individual needs | 0.75 | 0.46 | 0.78 | 0.43 | 0.83 | 0.87 | 0.65a |
aThese values indicated average variance extracted by each latent variable.
bThese values indicate that latent variable shared variance estimates exceed the average variance extracted of either or both variables.
cStatistically not significant interfactor correlation (
Estimated effects of administration format, site area, and health setting on the 7 eHealth literacy latent variables.
Scale | Admin formata,b | Site areaa,c | Health settinga,d |
1. Using technology to process health information |
|
0.02 (0.06) | 0.10 (0.06) |
2. Understanding of health concepts and language | −0.02 (0.07) | −0.00 (0.07) | 0.05 (0.08) |
3. Ability to actively engage with digital services |
|
−0.02 (0.05) | 0.07 (0.06) |
4. Feel safe and in control | −0.03 (0.06) | 0.12 (0.06) | −0.04 (0.07) |
5. Motivated to engage with digital services |
|
0.03 (0.06) | 0.10 (0.06) |
6. Access to digital services that work | 0.02 (0.06) | 0.02 (0.06) | −0.02 (0.07) |
7. Digital services that suit individual needs |
|
−0.01 (0.06) | 0.06 (0.07) |
aStandardized estimates reported; posterior SD for estimates shown in parentheses.
bAdministration format code: 0=interview, 1=paper.
cSite area code: 0=metropolitan, 1=regional.
dHealth setting code: 0=private clinic, 1=community health.
eItalicized values indicate statistically significant differences, significant if
The results of the 1-factor Bayesian models with all significant targeted factor loadings provided evidence of unidimensionality and local independence. Item invariance was supported by the testing of DIF for administration format, site area, and health setting. Measurement invariance across subgroups, including age, sex, education, language spoken at home, and information and communication technology use, was also established and reported by Cheng et al [
Visual inspection of the item characteristic curves showed distinct peaks for the response categories along the continuum of the latent trait for the most likely responses, indicating ordered thresholds for all items (
This study collected and examined validity evidence based on test content, response process, and internal structure of the eHLQ in the Australian community health setting. Items and formats were easy to read and use, and items were understood as intended. The Bayesian CFA and IRT analyses confirmed the robustness of the internal structure. However, discriminant validity based on estimates of the 7-factor BSEM was not well established and will require further investigation.
The cognitive interviews were successful in identifying 1 confusing term, which was revised, and the introductory page of the questionnaire was also improved. The results combined with observation during interviews and the survey as well as the limited number of missing values provided a wealth of information in support of the validity evidence on test content and response process of the eHLQ.
The final sample size of 525 of the cross-sectional survey provided an adequate sample size for both the CTT and IRT analyses. Although the sample had more women and university-educated participants than the Australian national averages, the sociodemographic characteristics of the participants still reflected a generally diverse sample. Nevertheless, a quarter of the sample did not use email or look for web-based information, showing that people with limited use of technology or eHealth were represented in the survey. This would ensure that the validity testing results of the eHLQ were also applicable to people with potentially lower eHealth literacy. In addition, the group differences in the 4 scales identified in the administration format for paper-based and interviews further pointed to the results that these 2 groups were significantly different in terms of age, education, and technology use. As such, the purpose of an interview option as a recruitment strategy to include older people or people with lower literacy was fulfilled. By contrast, evidence of measurement invariance across the 2 formats confirmed that responses were not influenced by interviewing bias or social desirability. Given a separate analysis of this sample found that older people also scored lower on the same 4 scales [
A rigorous assessment of the internal structure was undertaken using both the CTT and IRT approaches to ensure that different aspects of validity and reliability of the eHLQ data were investigated. For the CTT analysis, the Bayesian approach of applying informative priors was used. Although this modern approach may involve more steps in testing model fit, it allows for the hypothesis of approximate zeros for model parameters. Instead of being constrained to exact zeros, as in the traditional structural equation modeling approach, the approach provides a better approximation of the real world. As such, the seven 1-factor models were found to fit the data well, confirming scale homogeneity, while factor loadings and residual variances were acceptable. Estimates of internal consistency reliability were good for all scales, although scale 2 (
The characteristics of the test and items in accurately measuring eHealth literacy were further supported by the IRT analysis. The test and item characteristic curves demonstrated that participants with higher eHealth literacy were more likely to endorse items with agree and strongly agree. The information function curves indicated that the items could gather reliable and precise information across different levels of the underlying trait. Estimates further showed that the items had generally high sensitivity in discriminating participants with different levels of eHealth literacy. The item locations also supported the fact that the items represented different levels of difficulty. This is further verified with the item difficulty indexes from the CTT analysis, which showed the 2 estimates displaying very similar pattern, further strengthening the evidence that the items generally represented a range of difficulty levels of the latent factor. The use of MIMIC models also found no or negligible DIF for administration format, site area, and health setting, confirming measurement equivalence of the items across formats and settings.
Although it is noted that the Australian results are generally similar to the Danish validity testing results reported by Kayser et al [
Following the 1-factor models, a subsequent 7-factor model using informative priors for cross-loadings and residual covariances demonstrated excellent model fit of the factor structure, as hypothesized by the questionnaire developers. All target loadings were significant with acceptable factor loadings, and there was also no significant cross-loading for the chosen model of interest. Although the chosen model of interest has informative priors different from the chosen model of the Danish validity testing, the Australian data analyses generally replicate the Danish results, strengthening the evidence of the internal structure of the eHLQ.
A possible weakness in the psychometric properties of the eHLQ may be its discriminant validity. The AVE estimates suggested a lack of clear discrimination among all the scales except for scale 2 (
This study provided robust validity evidence of inferences drawn from the eHLQ when used in the diverse Australian community health settings. As this study was undertaken before the COVID-19 pandemic, which sees an increased acceptance and use of telehealth [
A possible limitation to the validity evidence is that the sample involved only participants who spoke and understood English well. Although the eHLQ is one of the first questionnaires developed simultaneously in 2 languages to minimize cultural references, both languages are from Western culture with generally well-developed national health care systems. How the psychometric properties perform in other cultural groups and countries is not clear. Future research on the eHLQ should include validity testing in cross-cultural settings including in different contexts and use. The Danish validity testing study was undertaken in the community setting involving the general population. However, this study only included people attending community health services. Future testing of the eHLQ in other Australian settings may strengthen the validity evidence of the tool for the general population.
The evidence presented in this study suggests that the eHLQ is a tool with robust psychometric properties. There is support for test content, and the items are understood as intended. Although there are potential weaknesses in discriminant validity, it is reasonable to suggest that the items can provide valid and reliable assessment of the 7 constructs of eHealth literacy in the diverse Australian health settings. The eHLQ is ready to be used to identify eHealth literacy strengths and challenges and assist the development of digital health interventions to ensure that people with limited digital access and skills are not being left behind.
Bayesian structural equation model of the eHealth Literacy Questionnaire with no prior. Output from Mplus.
Descriptive statistics of the eHealth Literacy Questionnaire items.
Psychometric properties of the eHealth Literacy Questionnaire single scales.
Bayesian model fit information of the eHealth Literacy Questionnaire 7-factor models.
Estimates for the direct effect of eHealth Literacy Questionnaire items on administration format, site area, and health setting.
Item characteristic curves and information function curves of the eHealth Literacy Questionnaire items (Item Response Theory for Patient-Reported Outcomes outputs).
average variance extracted
Bayesian structural equation modeling
confirmatory factor analysis
classical test theory
differential item functioning
eHealth Literacy Questionnaire
intraclass correlation coefficient
item response theory
multiple-indicators, multiple-causes
posterior predictive P
prior-posterior predictive P
The authors thank Dr Mukesh Haikerwal AC, Ms Jenny Ktenidis, Ms Rori Plaza of Altona North Medical Group and Cirqit Health, Ms Janine Scott, and Ms Olive Aumann of Carrington Health and Dr Ewa Piejko, Dr Adel Asaid, Dr Remon Eskander, and Dr Poate Radrekusa of St Anthony Family Medical Practice for their generous support. We also thank Prof Lisa Hanna of Deakin University for her input and Dr Polina Putrik, Visiting Fellow at Deakin University, for her assistance in data collection. RHO was funded in part through the National Health and Medical Research Council of Australia Principal Research Fellowship APP1155125.
None declared.