Published on 07.12.20 in Vol 22, No 12 (2020): December
Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/18613, first published Mar 11, 2020.
Psychometric Validation and Cultural Adaptation of the Simplified Chinese eHealth Literacy Scale: Cross-Sectional Study
Background: The rapid proliferation of web-based information on health and health care has profoundly changed individuals’ health-seeking behaviors, with individuals choosing the internet as their first source of information on their health conditions before seeking professional advice. However, barriers to the evaluation of people’s eHealth literacy present some difficulties for decision makers with respect to encouraging and empowering patients to use web-based resources.
Objective: This study aims to examine the psychometric properties of a simplified Chinese version of the eHealth Literacy Scale (SC-eHEALS).
Methods: Data used for analysis were obtained from a cross-sectional multicenter survey. Confirmatory factor analysis (CFA) was used to examine the structure of the SC-eHEALS. Correlations between the SC-eHEALS and ICEpop capability measure for adults (ICECAP-A) items and overall health status were estimated to assess the convergent validity. Internal consistency reliability was confirmed using Cronbach alpha (α), McDonald omega (ω), and split-half reliability (λ). A general partial credit model was used to perform the item response theory (IRT) analysis. Item difficulty, discrimination, and fit were reported. Item-category characteristic curves (ICCs) and item and test information curves were used to graphically assess the validity and reliability based on the IRT analysis. Differential item functioning (DIF) was used to check for possible item bias on gender and age.
Results: A total of 574 respondents from 5 cities in China completed the SC-eHEALS. CFA confirmed that the one-factor model was acceptable. The internal consistency reliability was good, with α=0.96, ω=0.92, and λ=0.96. The item-total correlation coefficients ranged between 0.86 and 0.91. Items 8 and 4 showed the lowest and highest mean scores, respectively. The correlation coefficients between the SC-eHEALS and ICECAP-A items and overall health status were significant, but the strength was mild. The discrimination of SC-eHEALS items ranged between 2.63 and 5.42. ICCs indicated that the order of categories’ thresholds for all items was as expected. In total, 70% of the information provided by SC-eHEALS was below the average level of the latent trait. DIF was found for item 6 on age.
Conclusions: The SC-eHEALS has been demonstrated to have good psychometric properties and can therefore be used to evaluate people’s eHealth literacy in China.
J Med Internet Res 2020;22(12):e18613
Currently, 4 billion people worldwide use the internet for various purposes every day, which increases their ability to search, digest, and use information in every aspect of their lives . The health care field cannot escape from this trend. The rapid proliferation of web-based information about health and health care has significantly changed individuals’ health-seeking behavior, with individuals choosing the internet as their first source of information regarding their health conditions before seeking professional advice [ , ].
Despite increased accessibility to relevant health information on the internet, searching, identifying, and using the right information is still challenging. Van der Vaart et al  found that easily identifiable web-based information may increase the discrepancies in health knowledge and choice of health care provider. Collecting web-based information is different from acquiring health-related information via traditional methods, such as hospital pamphlets, medical books, or professionals’ advice; acquiring useful and accurate web-based information necessitates specific skills [ , , ]. These skills include not only having professional knowledge about any given health conditions but also computer literacy, mobile phone literacy, and knowing how to navigate the internet [ , ]. Thus, eHealth literacy, including computer literacy and internet-related knowledge and skills, plays a key role in helping people search for web-based information and analyze, assess, and use that information to improve their health outcomes [ ]. eHealth information offers little value if the intended users lack the skills to effectively use this type of information [ ]. Measuring people’s eHealth literacy could help policy makers develop guidelines, strategies, and interventions to provide health information through the internet to people regarding different needs, preferences, and expectations in ways that they can understand and use it [ ].
In the last two decades, China has made remarkable progress in the development of internet networks and services. Currently, more than 800 million Chinese people actively use the internet every day . To maximize the effectiveness of the internet in improving people’s health, in 2018, the State Council of China announced an ambitious plan to develop an integral system to provide a broad spectrum of health care services through the internet across China in the next 10 years; this initiative intends to solve the long-term problems of kanbing nan and kanbing gui (too inaccessible and expensive to visit physicians) [ ]. An instrument that comprehensively evaluates people’s skills and literacy to understand and use web-based information and services is essential. In particular, given that China has a dual social urban-rural system, the gap between the cities and the countryside, including economic, educational, health care, and so on, was and continues to be tremendous. The provision of consumer-tailored eHealth information that meets their levels of eHealth literacy is the key to engaging them in making sound health decisions [ ].
At present, few measures exist to specifically measure eHealth literacy, which has a different conceptual structure compared with traditional health literacy. In 2006, the eHealth literacy scale (eHEALS) was developed to focus on assessing the skills for finding, evaluating, and applying web-based health information to improve one’s health outcomes . It has been translated and validated in several countries and has been demonstrated to be a valuable instrument for assessing the effects of eHealth literacy on helping decision makers develop appropriate strategies to provide web-based information to improve people’s health.
In China, several studies have reported the performance of eHEALS in different settings, and all of them showed some limitations. The generalizability of the findings from the study by Chang and Schultz  was questionable because they used a total of 352 patients from one selected hospital from a city in China, and their targeted populations were only those who reported having chronic conditions. For the study by Ma and Wu [ ], the major limitation was that all the respondents included in that study were rural residents and were recruited from one district of a small city in Western China (the poorest part of China). By 2018, approximately 60% of the population in China lived in urban areas; therefore, assessing the psychometric properties of the eHEALS on only rural residents from one district would lead to ubiquitous selection and information bias [ ]. Furthermore, the study by Zibrik et al [ ] also reported the psychometric properties of the eHEALS; however, their study was not conducted in mainland China, and the targeted population was British Columbia’s immigrant Chinese and Punjabi senior population. The study by Guo et al [ ] seemed to be the first psychometric study of the Chinese eHEALS; however, 2 issues need to be addressed. First, their work was presented in Chinese rather than English, where the results are difficult for non-Chinese researchers to understand—this limits international or regional comparisons within Asian countries or between Chinese populations from different countries. Second, the targeted population of that study was 110 high school students recruited from Beijing (the most developed region in China). The selection and information bias could not be neglected. Moreover, except for the study by Ma and Wu [ ], no previous studies used both classical test theory (CTT) and item response theory (IRT) to assess the validity of the Chinese eHEALS, and no study has adopted the differential item functioning (DIF) analysis to investigate the item variance of the scale. Considering the limitations summarized above, it is important to assess the performance of the Chinese version of eHEALS using more comprehensive methods and a better representative sample, which covers respondents from developed and underdeveloped areas, urban and rural residents, and across a wide age range.
Aims of the Study
This study aims to examine the psychometric properties of a simplified Chinese version of the eHEALS (SC-eHEALS) in the Chinese population based on a multicenter cross-sectional study.
The translation of the SC-eHEALS followed the international protocol . Two translators, who were native Chinese speakers but were fluent in English, were invited to translate the eHEALS from English into simplified Chinese independently. The 2 transcripts were collected by the local research team for discussion and synthesis. Subsequently, a revised version was produced and sent to 2 professional translators for independent backward translation. The local research team examined the back translation against the original English version to identify any discrepancies, addressed the disputed items, and refined the translation, focusing on cultural adaptation until consensus was achieved by all the research team members. Then, a cognitive debriefing was carried out that involved 15 patients recruited from a hospital in Guangzhou. All the respondents were asked to comment on the response options and any word or phrasing that they found difficult to understand. Respondents were asked to describe in their own language what the word or phrasing meant to them. The results of the pilot study were discussed by the local research team, and then the final version of the SC-eHEALS was verified.
Study Design and Data Collection
Data used in this study were obtained from a multicenter cross-sectional survey that investigated patients’ attitudes toward patient-centered care (PCC) in China from November 2019 to January 2020. Patients were recruited from the inpatient departments of 8 hospitals from 5 cities (Guangzhou, Shenzhen, Zhanjiang, Meizhou, and Shaoguan) of the Guangdong province (Southern China). All patients from the target hospitals during the appointed survey period were invited to participate in the survey. The inclusion criteria were as follows: (1) aged ≥18 years, (2) can read and speak Chinese, (3) no cognitive problems, and (4) able to provide informed consent. A team of investigators with rich experience in conducting face-to-face interviews was recruited to carry out the fieldwork. With the assistance of ward nurses, all respondents were asked to complete a structured questionnaire that included questions about demographic characteristics, socioeconomic status (SES), health conditions, well-being, health services use, lifestyle, and attitudes toward PCC. To conduct confirmatory factor analysis (CFA), the minimum sample size required was 300 [, ]. For the IRT analysis, the preferred sample size required is 500 for analyzing a scale comprising polytomous items [ ]. In total, the data of 574 patients who completed the SC-eHEALS questionnaire were elicited for our psychometric analysis. Informed consent was obtained from all respondents. The study protocol was approved by the institutional review board of the second affiliated hospital of Guangzhou Medical University (ethical approval ID: 2019-ks-28).
eHEALS is used to measure consumers’ combined knowledge, comfort, and perceived skills at finding, evaluating, and applying eHealth information to manage health problems . It was developed based on a framework that consists of 6 dimensions to understand and use eHealth information [ ]. The eHEALS has 8 items that are rated on a 5-point Likert scale ranging from strongly disagree to strongly agree. A one-factor structure was confirmed in the original study, and the reliability was acceptable with a Cronbach alpha of .88 [ ]. The sum score of eHEALS ranges from 8 to 40, where a higher score indicates greater perceived eHealth literacy. The translation of eHEALS to SC-eHEALS has been discussed earlier.
The ICEpop CAPability Measure for Adults
The ICEpop capability measure for adults (ICECAP-A) is a generic and preference-based instrument that evaluates an individual’s well-being . It has 5 items (stability, attachment, autonomy, achievement, and enjoyment), and each item has 4 response options that range from fully capable to not capable.
Overall Health Status
The respondents’ overall health status was evaluated using a visual analogue scale (VAS). They were presented with a scale ranging from 0 to 100, where 0 represents the worst health status and 100 represents the best health status they can imagine on the surveying day.
CFA was used to examine the structure of SC-eHEALS. The model fit was determined by 4 indicators, that is, root mean square error of approximation (RMSEA≤0.08, fair), the comparative fit index (CFI>0.95, good), the Tucker-Lewis index (TLI>0.95, good), and standardized root mean square residual (SRMR<0.08, acceptable) . Moreover, the Akaike information criterion (AIC) and Bayesian information criterion (BIC) were supplemented to compare the performance of the models, with smaller values indicating a better fit. Exploratory factor analysis (EFA) was used when the one-factor model reported in the original study was not supported by CFA.
Convergent validity was confirmed by evaluating the correlations between the SC-eHEALS and ICECAP-A items and the VAS item. Pearson correlation coefficient (r) was used to assess the strength of the correlation with r>0.3 identified as moderate .
One-way analysis of variance (ANOVA) was used to assess the known-group validity of the SC-eHEALS. As individuals’ levels of eHealth literacy were impossible to observe directly, on the basis of literature review, we hypothesized that the patients who were young, highly educated, or fully employed (representing good SES) would show a high degree of eHealth literacy.
Item Statistics and Internal Consistency Reliability
The mean and SD of the SC-eHEALS scores at both the item- and scale-level were reported as well as the ceiling and floor effects. The internal consistency reliability was estimated using Cronbach alpha (α>.7, acceptable), McDonald omega (ω>0.7, acceptable), and split-half reliability (λ>0.7, acceptable). Item-total correlations (>0.7 acceptable) and alpha were presented if items were deleted .
Considering that there is no gold standard for model selection to perform IRT analysis, we adopted 3 models in this study, compared their performance, and reported the results of the best-fit model. The first model was the partial credit model (PCM), which is an extended version of the Rasch model. The second model was the general partial credit model (GPCM), which is a modified version of PCM and used to estimate 2 parameters in the analysis; and the last was the rating scale model, which is another Rasch family model that requires all items to have the same number of options. The details of the IRT models can be found in the study by DeMars . The likelihood ratio test was used to compare the performance of the models. AIC and BIC were also used to assess the model fit. The results of model comparisons showed the GPCM outperformed the other 2 models [ ]. For GPCM analysis, the (1) discrimination, (2) difficulty, and (3) item fit (the value of S-χ2) were calculated for each item, along with the item information curves (IICs) and test information curve (TIC) [ ]. The item-category characteristic curves (ICCs) graphically presented the probability of the response to a given item in each category as the function of the latent trait (eHealth literacy).
Differential Item Functioning
DIF was used to check for the possible item bias caused by responses from different subgroups (gender, age, family registry, chronic conditions, and educational level) in the sample . Pseudo McFadden R2 was used to estimate the magnitude of the DIF. R2<0.13 was deemed as negligible, 0.13 to 0.26 as moderate, and >0.26 as large. [ ]
R (R foundation) was used for data analysis, and the P value was set at <.05. For IRT analysis, the Bonferroni correction was applied, and the P value was set at <.006 (.05/8).
In this study, more than half of the respondents were men (292/574, 50.9%), and the mean age was 45.58 years (SD 16.18). Almost half of them completed tertiary educational attainment (263/574, 45.8%), and 50.8% (231/574) were urban residents. Nearly half of the respondents reported living with chronic conditions (281/574, 49.1%) and only one fifth of the respondents reported that their disease was a severe threat to their lives (112/574, 19.8%;).
|Sex, n (%)|
|Age (years), mean (SD)||45.58 (16.18)|
|Education, n (%)|
|No or primary||93 (16.2)|
|Secondary or postsecondary||218 (37.9)|
|Tertiary or above||263 (45.8)|
|Family registry, n (%)|
|Caregiver, n (%)|
|Living status, n (%)|
|Live alone||57 (9.9)|
|Live with family or others||517 (90.1)|
|Working status, n (%)|
|Income levela(CNY ¥ [US $]), n (%)|
|≤900b (64.3)||112 (19.6)|
|901-1800 (64.4-128.6)||48 (8.4)|
|1801-2700 (128.6-192.9)||68 (11.9)|
|2701-3800 (192.9-271.4)||78 (13.7)|
|3801-6400 (271.4-457.1)||125 (21.9)|
|≥6401 (457.1)||140 (24.5)|
|BMI, n (%)|
|Chronic condition, n (%)|
|Self-reported health condition, n (%)|
|Severe threat to life||112 (19.8)|
|Moderate threat to life||112 (19.8)|
|Mild threat to life||137 (24.2)|
|No threat to life||206 (35.8)|
aDisposable personal income per month.
b100 CNY ¥ equals approximately 14 US $ (February 2020 exchange rate).
The results of the CFA showed that the model fit of the one-factor model was not satisfactory with a low TLI and high RMSEA values (). We further examined items with high residual correlations and modification indices. On examining the wordings and polychoric correlation of items, we found that the nonrandom measurement error was caused by item redundancy. The fit of the revised model was improved after we modified the model and specified the error covariance between items 1 and 2, 3 and 4, and 7 and 8. A two-factor structure was suggested by the EFA ( ). The CFA indicated that the performance of the revised two-factor model was not better than the revised one-factor model. After checking the item correlations and residuals, a revised two-factor model was confirmed, with RMSEA=0.08, SRMR=0.02, CFI=0.99, and TLI=0.96. The revised two-factor model outperformed the other models. The standardized factor loadings for the observed variables of all CFA models are presented in .
|Model||Model fit statistics|
|χ2 (df)||P value||RMSEAa (95% CI)||CFIb||TLIc||SRMRd||AICe||BICf|
|One-factor||443.56 (20)||<.001||0.19 (0.18-0.21)||0.92||0.88||0.04||7905.29||7974.93|
|Revised one-factor||134.76 (17)||<.001||0.11 (0.09-0.12)||0.97||0.96||0.03||7602.49||7685.19|
|Two-factor||252.73 (19)||<.001||0.15 (0.13-0.16)||0.95||0.93||0.03||7716.46||7790.45|
|Revised two-factor||83.2 (17)||<.001||0.08 (0.06-0.1)||0.99||0.98||0.02||7550.93||7633.63|
aRMSEA: root mean square error of approximation.
bCFI: comparative fit index.
cTLI: Tucker-Lewis index.
dSRMR: standardized root mean square residual.
eAIC: Akaike information criterion.
fBIC: Bayesian information criterion.
Convergent and Known-Group Validity
shows the correlations between the SC-eHEALS and ICECAP-A items and overall health status. All the correlations of SC-eHEALS with the other measures were statistically significant, but the value of coefficients indicated a mild correlation. The results of ANOVA indicate that respondents who were young, highly educated, and fully employed tended to report a high level of eHealth literacy, which indicated that the known-group validity of the SC-eHEALS was satisfied ( ).
Chinese eHealth literacy scale itemsa
|Stability||P value||Attachment||P value||Enjoyment||P value||Achievement||P value||Autonomy||P value||Value||P value|
aFor the eHealth literacy scale, a higher score indicates good eHealth.
bFor ICECAP-A, a higher score indicates worse status of well-being.
cVAS: visual analogue scale of overall physical health.
deHEALS: eHealth literacy scale.
|Characteristics||Mean (SD)||P value|
|No or primary||28.29 (7.99)|
|Secondary or postsecondary||28.46 (7.05)|
|Tertiary or above||30.27 (5.93)|
Item Statistics and Internal Consistency Reliability
shows that item 8 was rated as the most difficult item with a mean score of 3.49 (SD 1.01), whereas item 4 was rated as the easiest item with a mean score of 3.75 (SD 0.93). Items showed no floor effects, but some ceiling effects, which ranged from 16.9% to 19.69%. The reliability of the scale was good, as the Cronbach alpha was .96, McDonald omega was 0.92 and split-half reliability was 0.96. In addition, the value of alpha if an item was deleted was lower than that observed if the item was retained, and the item-total correlation coefficients ranged between 0.86 and 0.91. The SC-eHEALS scores stratified according to respondents’ characteristics are presented in .
|SC-eHEALSa||Response, n||Item statistics||Internal consistency reliability|
|Strongly disagree||Disagree||Neutral||Agree||Strongly agree||Mean (SD)||Floor, n (%)||Ceiling, n (%)||Alpha if item deletedb||Item-total correlationb|
|eHEALSc 1||18||36||150||267||103||3.70 (0.94)||18 (3.14)||103 (17.94)||.88||0.91|
|eHEALS 2||16||29||156||264||109||3.73 (0.92)||16 (2.79)||109 (18.99)||.88||0.91|
|eHEALS 3||14||39||141||278||102||3.72 (0.92)||14 (2.44)||102 (17.77)||.82||0.86|
|eHEALS 4||15||37||135||274||113||3.75 (0.93)||15(2.61)||113(19.69)||.86||0.89|
|eHEALS 5||17||49||160||242||106||3.65 (0.97)||17 (2.96)||106 (18.47)||.87||0.90|
|eHEALS 6||17||45||169||242||101||3.64 (0.96)||17 (2.96)||101 (17.60)||.86||0.89|
|eHEALS 7||20||47||195||208||104||3.57 (0.99)||20 (3.48)||104 (18.12)||.84||0.88|
|eHEALS 8||22||58||205||192||97||3.49 (1.01)||22 (3.83)||97 (16.90)||.81||0.86|
|Overall||N/Ad||N/A||N/A||N/A||N/A||29.26 (6.78)e||8 (1.57)||40 (10.63)||N/A||N/A|
aSC-eHEALS: simplified Chinese eHealth literacy scale.
bCronbach alpha=.96; McDonald omega=.92; split-half reliability=.96.
ceHEALS: eHealth literacy scale.
dN/A: not applicable.
eThe overall score of the eHEALS ranges between 0 and 40.
The results of the GPCM analysis are presented in. The discrimination indices of items ranged between 2.63 and 5.42, which indicated that the SC-eHEALS can distinguish between individuals with either low or high eHealth literacy, corresponding with the latent trait sensitivity. The item difficulty of threshold 1 and threshold 4 ranged from −2.03 to −1.79 and 0.84 to 1.04, respectively. The P value of S-χ2 (Specific item-fit index) reflected that items 3 (P<.001), 4 (P=.003), and 8 (P<.001) might show misfit to some extent. Most of the information (70%) provided by the SC-eHEALS was between −6 and 0 on the latent trait scale (the comparisons between different IRT models are presented in , and the misfit items of the SC-eHEALS are presented in ).
|SC-eHEALSa||Discrimination of item||Difficulty of item||Specific item-fit index, S-χ2 (df)||P valueb||Information (0-100c)|
|T1d||T2e||T3f||T4g||−6 to 0||0 to 6|
|eHEALS 1||5.27||−1.79||−1.27||−0.41||0.90||19.041 (17)||.33||72.65||27.35|
|eHEALS 2||5.42||−1.85||−1.37||−0.42||0.84||24.171 (16)||.09||72.95||27.05|
|eHEALS 3||3.09||−2.03||−1.37||−0.51||1.01||61.57 (21)||<.001||71.62||28.38|
|eHEALS 4||4.17||−1.94||−1.31||−0.50||0.85||38.969 (18)||.003||72.95||27.05|
|eHEALS 5||4.52||−1.87||−1.17||−0.33||0.89||32.962 (19)||.05||70.73||29.27|
|eHEALS 6||3.83||−1.89||−1.24||−0.31||0.96||27.916 (20)||.11||69.67||30.33|
|eHEALS 7||3.24||−1.80||−1.26||−0.17||0.95||41.64 (24)||.01||66.58||33.42|
|eHEALS 8||2.63||−1.80||−1.20||−0.06||1.04||73.951 (25)||<.001||64.01||35.99|
aSC-eHEALS: simplified Chinese eHealth literacy scale.
bItems with P<.05/8=.0062 were flagged as potentially misfit.
c0 means no information, 100 means full information.
dT1: threshold categories 1 and 2.
eT2: threshold categories 2 and 3.
fT3: threshold categories 3 and 4.
gT4: threshold categories 4 and 5.
hN/A: not applicable.
The ICCs and IICs for SC-eHEALS are graphically presented inand , respectively. The curves of the ICCs showed that the order of categories’ thresholds for all the items was as expected, which meant that all categories were adequate in terms of placing a respondent on the scale. The distributions of the IICs were multimodal. The shapes of items 1 and 2 were the steepest and provided more information than the other items. The shapes of items 3 and 8 were the flattest, which means little information was provided by these items. TIC of the SC-eHEALS is presented in .
Item-person map displays the location parameters of items as well as the distribution of person parameters along the same latent dimension. We found that the gap between threshold 3 and threshold 4 was larger than the other gaps; however, the discrepancy was diminished from item 1 to 8. In contrast, the gap between threshold 1 and threshold 2 was smaller than the others; however, the discrepancy increased from items 1 to 8. The distribution of person parameters showed some respondents located at the high end of the latent trait scale, which means they are very likely equipped with strong eHealth literacy ().
Item 6 of the SC-eHEALS showed a uniform DIF when considering the respondents from different age groups. However, the magnitude was smaller than 0.13, confirming that the effect size of the DIF was negligible  ( ).
In China, an increasing number of people have turned to the internet to seek health care information because of the rapid proliferation of web-based medical information. However, whether users can leverage such information to improve their health can hardly be measured. The findings of this study support that the SC-eHEALS () is a valid and reliable instrument for measuring self-reported eHealth literacy in China. For the general public, the SC-eHEALS provides a measure to help them evaluate web-based health information and critically appraise the quality of eHealth resources, which could protect them from the harm of low-quality information and empower them to manage their health. Policy makers provide important information for understanding the population’s perceived degree of eHealth literacy and developing cost-effective strategies for upgrading the medical care sector by leveraging the internet [ ].
Comparisons With Previous Studies
The original one-factor structure of the eHEALS, as confirmed by previous studies in mainland China [, ], was acceptable, but was not fully supported by our study. This was not an unforeseen result; some other studies have reported a two-factor [ , ] or a three-factor structure [ , ]. Diviani et al [ ] pinpointed that when using CFA, neither the single-factor model, originally proposed by the developers, nor the two-factor model, suggested by other research, showed an adequate fit to the data. Cummings [ ] indicated that translation might change the original meaning of the items, and these changes could affect perceived meanings for targeted respondents. Furthermore, the original eHEALS was developed at a time before the rise of social media, which totally changed people’s interaction with health information, which might affect the structure of the eHEALS [ ]. Furthermore, the eHEALS was developed based on a model that originally consisted of 6 domains of literacy, and Noman et al [ ] suggested that each skill would require independent measurement. However, in this study, the two-factor structure was not without problems. First, the values of the CFI, TLI, and SRMR indicated that the performance of the revised one-factor model was not worse than that of the two-factor model. Only the RMSEA value supported the revised two-factor model outperforming the other models. A similar finding was reported in the study by Paige et al [ ], in which they finally confirmed the one-factor structure. Second, the factor loadings of each item in the revised two-factor model were similar to the one-factor model; item 7 even showed a lower value (0.594), which indicated that the difference in stability between the 2 structures was negligible. Given that people’s different abilities to manipulate web-based information is attributed to their demographics, SES, and health conditions, we decided to maintain the one-factor structure. Studies exploring the structure of the SC-eHEALS in other subpopulations should be carried out in the future. A clear structure of the SC-eHEALS would be useful in facilitating the computer adaptive test (CAT) in future practice. Administering the SC-eHEALS based on CAT can strengthen its precision and sensitivity [ ], which ensures that the chosen items can meet the respondents’ eHealth literacy levels.
The proportion of respondents choosing the options of neutral and above was high in this study, which might indicate that most of them are equipped with middle-to-high eHealth literacy and skills and are confident in searching, understanding, analyzing, and using eHealth information. The ceiling effect of all the items of the SC-eHEALS was detected, despite the strength of the effect not being very strong. The distribution of the responses in eHEALS was not reported by other studies in China. However, Paige et al  reported that the mean score of the eHEALS ranged between 3.57 and 3.96 in the US population. It remained that the discriminant ability of the SC-eHEALS might not be strong enough to differentiate people with different levels of eHealth literacy. In addition, the SC-eHEALS showed an excellent reliability with a Cronbach alpha (.96) higher than that reported by other studies conducted in China [ , ] and for some other language versions [ , , , , ]. However, Chang and Schultz [ ] found that removing items 7 or 8 could improve the reliability of the eHEALS in China, which was not reported by any other study. Moreover, we found that the mean score of the eHEALS varied widely across different populations. For example, Diviani et al [ ] reported the same findings as ours, as respondents rated item 4 as the easiest and item 8 as the most difficult. Chung and Nahm [ ] found that item 4 was perceived as the easiest item, whereas item 7 was perceived as the most difficult item by a US sample. Van der Vaart et al [ ] identified that items 3 and 8 received the highest and lowest mean scores of all items, respectively. Given that different studies have recruited samples with different subpopulations and that the proliferation of the internet in different countries is varied, further exploration should be carried out to draw more international comparisons.
The GPCM was demonstrated to be the best-fit IRT model in this study. No previous studies used the GPCM to test the psychometric properties of eHEALS. Diviani et al  used the nonparameter IRT model, Paige et al [ ] used the PCM model, and Zrubka et al [ ] and Ma and Wu [ ] used the graded response model. Our study enriches the knowledge of the application of IRT models and supports existing research on the psychometric analysis of eHEALS using IRT methods. GPCM analysis showed that the overall performance of SC-eHEALS was satisfactory. ICCs indicated that the response categories of each item were ordered, and all categories were most probably at the same point on the continuum. For the location parameter, all items were placed between −3 and 3 across the scale of the latent trait. The discrimination of the items was positive and could discriminate between individuals with different levels of eHealth literacy. The item-person map demonstrated that the items of the SC-eHEALS were located within a reasonable range and were well centered with respect to the person measure distribution. Diviani et al [ ] reported similar findings in the Italian eHEALS; however, Paige et al [ ] showed that the response categories of the eHEALS covered a wider range of latent traits. The information curves showed a multimodal distribution of rural Chinese citizens [ ] and the Hungarian population as shown in the study by Zrubka et al [ ]. The TIC of the SC-eHEALS provided the precise estimation near the center of the ability scales, which ranged between −3 and 2, but as the ability level approached the extremes of the scale, the accuracy of the test decreased significantly. The plot of misfit items showed that the majority of imprecise estimations occurred at the options of neutral and agree, indicating that these 2 options might not be accurate enough to measure the latent trait among people with moderate-to-high eHealth literacy. To yield a more precise measure of eHealth literacy, future studies should explore what response options are more appropriate to be included in the SC-eHEALS [ ].
Moreover, item 6 (skills to evaluate health resources) was labeled as having DIF among respondents of different age groups (older vs younger), which reflected that they have an unequal probability of giving a response to this item. No previous study reported a similar finding at the item level using DIF analysis. Ma and Wu  indicated that item 6 has the lowest discriminant power in Chinese eHEALS. However, caution should be exercised when drawing this conclusion, as we cannot determine whether the DIF occurred due to a form of measurement error [ ]. In other words, we can neither unequivocally infer that the DIF of item 6 reflected real group differences of the underlying trait among respondents nor attribute this finding to bias during the process of measurement.
Strengths and Limitations
This study has several strengths. First, the respondents who participated in this study came from both developed and underdeveloped areas, were urban and rural residents, and spanned a wide age range, showing better representativeness than the previous studies in China. Second, this study assessed the psychometric properties of SC-eHEALS using both CTT and IRT methods. Third, for IRT analysis, the best-fit model was selected based on comparison of the performance of 3 IRT models. Fourth, DIF analysis was first used to evaluate the item bias and variance of the eHEALS in the Chinese population. The comparisons between the characteristics of the Chinese eHEALS validation studies are presented in.
Several limitations of this study should be addressed. First, all respondents were from inpatient departments. This means that most of them might have poor health status, which posed some degree of selection bias. Patients with mild or no health problems should be included in follow-up studies to further test the psychometric properties of SC-eHEALS. Second, the sample size of this study was less than 1000, which might generate some uncertainties in estimating the IRT model, especially using the GPCM (a two-parameter model). A larger sample should be collected in future studies to validate our findings. Third, we did not differentiate between patients with different diseases when evaluating the psychometric properties of the SC-eHEALS, which might have created some problems in explaining the validity of the instrument. The performance of SC-eHEALS should be further assessed in distinct patient groups. Fourth, we did not collect information on respondents’ behaviors related to their internet use, such as the frequency and types of websites; thus, criterion validity cannot be assessed. Finally, the information of patients who refused to participate in the survey was not recorded, which might have led to a measure of information bias.
This study evaluated the psychometric properties of SC-eHEALS using a large sample of patients from a multicenter study in China. It provides empirical evidence that SC-eHEALS is a valid, reliable, and parsimonious instrument for evaluating the degree of eHealth literacy in Chinese people with different demographics, SES (eg, rural and urban residents), and health status. CFA did not fully support the original one-factor structure, and further exploration is needed. IRT analysis suggested that SC-eHEALS might not be effective for use in people with very high or low eHealth literacy. Further studies are needed to detect the heterogeneity of the SC-eHEALS in different subpopulations and further assess its criterion validity.
All authors contributed to the concept and study design. RX and SL prepared the translation, RX conducted the data analysis, and RX and LZ drafted the manuscript. All authors critically revised the manuscript. WD and EW supervised the study. All authors approved the final manuscript for publication.
Conflicts of Interest
The results of the exploratory factor analysis.DOCX File , 20 KB
The standardized factor loadings for the observed variables of all confirmatory factor analysis models.DOCX File , 23 KB
The simplified Chinese eHealth literacy scale scores stratified according to respondents’ characteristics.DOCX File , 21 KB
The comparisons between different item response theory models.DOCX File , 20 KB
The misfit items of the simplified Chinese eHealth literacy scale.PNG File , 682 KB
Test information curve of the simplified Chinese eHealth literacy scale.PNG File , 131 KB
Results of differential item functioning analysis.DOCX File , 20 KB
The Chinese version of the eHealth literacy scale.DOCX File , 22 KB
Summary of the Chinese eHEALS studies.DOCX File , 21 KB
- Richtering SS, Morris R, Soh S, Barker A, Bampi F, Neubeck L, et al. Examination of an eHealth literacy scale and a health literacy scale in a population with moderate to high cardiovascular risk: Rasch analyses. PLoS One 2017;12(4):e0175372 [FREE Full text] [CrossRef] [Medline]
- Quinn S, Bond R, Nugent C. Quantifying health literacy and eHealth literacy using existing instruments and browser-based software for tracking online health information seeking behavior. Comput Hum Behav 2017 Apr;69:256-267. [CrossRef]
- Paige SR, Krieger JL, Stellefson M, Alber JM. eHealth literacy in chronic disease patients: an item response theory analysis of the eHealth literacy scale (eHEALS). Patient Educ Couns 2017 Feb;100(2):320-326 [FREE Full text] [CrossRef] [Medline]
- van der Vaart R, van Deursen AJ, Drossaert CH, Taal E, van Dijk JA, van de Laar MA. Does the eHealth Literacy Scale (eHEALS) measure what it intends to measure? Validation of a Dutch version of the eHEALS in two adult populations. J Med Internet Res 2011 Nov 9;13(4):e86 [FREE Full text] [CrossRef] [Medline]
- Silver MP. Patient perspectives on online health information and communication with doctors: a qualitative study of patients 50 years old and over. J Med Internet Res 2015 Jan 13;17(1):e19 [FREE Full text] [CrossRef] [Medline]
- Karnoe A, Furstrand D, Christensen KB, Norgaard O, Kayser L. Assessing competencies needed to engage with digital health services: development of the ehealth literacy assessment toolkit. J Med Internet Res 2018 May 10;20(5):e178 [FREE Full text] [CrossRef] [Medline]
- Parker S, Prince A, Thomas L, Song H, Milosevic D, Harris MF. Electronic, mobile and telehealth tools for vulnerable patients with chronic disease: a systematic review and realist synthesis. BMJ Open 2018 Aug 29;8(8):e019192. [CrossRef] [Medline]
- Neter E, Brainin E. Perceived and performed ehealth literacy: survey and simulated performance test. JMIR Hum Factors 2017 Jan 17;4(1):e2. [CrossRef] [Medline]
- Norman CD, Skinner HA. Ehealth literacy: essential skills for consumer health in a networked world. J Med Internet Res 2006 Jun 16;8(2):e9 [FREE Full text] [CrossRef] [Medline]
- China’s H2 2018 Internet User Base Reaches 829 million. China Telecom. 2019. URL: https://www.marbridgeconsulting.com/marbridgedaily/archive/article/114539/chinas_h2_2018_internet_user_base_reaches_829_mln [accessed 2019-02-01]
- Development and Promotion of 'Internet + Healthcare' in China. State Council of China. 2018. URL: http://www.gov.cn/zhengce/content/2018-04/28/content_5286645.htm [accessed 2019-02-01]
- Norman CD, Skinner HA. eHEALS: The eHealth Literacy Scale. J Med Internet Res 2006 Nov 14;8(4):e27 [FREE Full text] [CrossRef] [Medline]
- Chang A, Schulz P. The measurements and an elaborated understanding of Chinese ehealth literacy (C-EHEALS) in chronic patients in China. Int J Environ Res Public Health 2018 Jul 23;15(7):- [FREE Full text] [CrossRef] [Medline]
- Ma Z, Wu M. The Psychometric Properties of the Chinese eHealth Literacy Scale (C-eHEALS) in a Chinese Rural Population: Cross-Sectional Validation Study. J Med Internet Res 2019 Oct 22;21(10):e15720 [FREE Full text] [CrossRef] [Medline]
- Zibrik L, Khan S, Bangar N, Stacy E, Novak Lauscher H, Ho K. Patient and community centered eHealth: exploring eHealth barriers and facilitators for chronic disease self-management within British Columbia’s immigrant Chinese and Punjabi seniors. Health Policy and Technology 2015 Dec;4(4):348-356. [CrossRef]
- Guo S, Yu X, Sun Y, Nie D, Li X, Wang L. eHEALS. Zhongguo jian kang jiao yu 2013;29:106-108.
- Wild D, Grove A, Martin M, Eremenco S, McElroy S, Verjee-Lorenz A, ISPOR Task Force for TranslationCultural Adaptation. Principles of good practice for the translation and cultural adaptation process for patient-reported outcomes (PRO) measures: report of the ISPOR task force for translation and cultural adaptation. Value Health 2005;8(2):94-104 [FREE Full text] [CrossRef] [Medline]
- Floyd FJ, Widaman KF. Factor analysis in the development and refinement of clinical assessment instruments. Psychol Assess 1995;7(3):286-299. [CrossRef]
- Klein B. Principles and Practice of Structural Equation Modeling. Fourth Edition. New York, USA: The Guilford Press; 2015.
- Linacre J. Sample Size and Item Calibration or Person Measure Stability. Rasch Meas Trans. 1994. URL: http://www.rasch.org/rmt/rmt74m.htm [accessed 2019-02-01]
- Al-Janabi H, Flynn TN, Coast J. Development of a self-report measure of capability wellbeing for adults: the ICECAP-A. Qual Life Res 2012 Mar;21(1):167-176 [FREE Full text] [CrossRef] [Medline]
- Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Eq Model: Multidiscip J 1999 Jan;6(1):1-55. [CrossRef]
- Xie Y, DeVellis RF. Scale development: theory and applications. Contemp Sociol 1992 Nov;21(6):876. [CrossRef]
- DeMars C. Item Response Theory. Oxford, UK: Oxford University Press; 2010.
- Coulacoglou C. Psychometrics and Psychological Assessment: Principles and Applications. London, UK: Academic Press; 2017.
- Valdez JP, Chu SK. Examining the psychometric validity of the five-item gratitude questionnaire: an item response theory approach. J Psychoeduc Assess 2018 Dec 17;38(4):529-536. [CrossRef]
- Xu RH, Cheung AW, Wong EL. Development and validation of an instrument to measure patient engagement in Hong Kong Special Administrative Region, China. Patient Prefer Adherence 2018 Sep;Volume 12:1667-1675. [CrossRef]
- Zumbo A. A Handbook on the Theory and Methods of Differential Item Functioning (DIF): Logistic Regression Modeling as a Unitary Framework for Binaryand Likert-type (Ordinal) Item Scores. Semantic Scholar. 1999. URL: https://www.semanticscholar.org/paper/A-Handbook-on-the-Theory-and-Methods-of-Item-(DIF)-Zumbo/7f88fb0ad98645582665532600d7c46406fa2db6 [accessed 2019-02-01]
- Lipovetsky S. Modern psychometrics With R. Technometrics 2020 Jan 31;62(1):135-137. [CrossRef]
- Neter E, Brainin E, Baron-Epel O, editors. The third digital divide in the health domain: is internet use for health purposes associated with health benefits? In: EHealth: Current Evidence, Promises, Perils, and Future Directions. Bingley, UK: Emerald Publishing Limited; 2018.
- Soellner R, Huber S, Reder M. The concept of ehealth literacy and its measurement. J Media Psychol 2014 Jan;26(1):29-38. [CrossRef]
- Sudbury-Riley L, FitzPatrick M, Schulz PJ. Exploring the measurement properties of the ehealth literacy scale (EHEALS) among baby boomers: a multinational test of measurement invariance. J Med Internet Res 2017 Feb 27;19(2):e53 [FREE Full text] [CrossRef] [Medline]
- Diviani N, Dima AL, Schulz PJ. A psychometric analysis of the Italian version of the ehealth literacy scale using item response and classical test theory methods. J Med Internet Res 2017 Apr 11;19(4):e114 [FREE Full text] [CrossRef] [Medline]
- Cross-cultural survey methods. Technometrics 2003 May;45(2):182-182. [CrossRef]
- Zrubka Z, Hajdu O, Rencz F, Baji P, Gulácsi L, Péntek M. Psychometric properties of the Hungarian version of the eHealth Literacy Scale. Eur J Health Econ 2019 Jun;20(Suppl 1):57-69 [FREE Full text] [CrossRef] [Medline]
- Chung S, Nahm E. Testing reliability and validity of the eHealth Literacy Scale (eHEALS) for older adults recruited online. Comput Inform Nurs 2015 Apr;33(4):150-156 [FREE Full text] [CrossRef] [Medline]
- Santelices MV, Wilson M. On the relationship between differential item functioning and item difficulty. Educ Psychol Measur 2011 Aug 4;72(1):5-36. [CrossRef]
|AIC: Akaike information criterion|
|ANOVA: one-way analysis of variance|
|BIC: Bayesian information criterion|
|CAT: computer adaptive test|
|CFA: confirmatory factor analysis|
|CFI: comparative fit index|
|CTT: classical test theory|
|DIF: differential item functioning|
|EFA: exploratory factor analysis|
|eHEALS: eHealth literacy scale|
|GPCM: general partial credit model|
|ICC: item-category characteristic curve|
|ICECAP-A: ICEpop capability measure for adults|
|IIC: item information curve|
|IRT: item response theory|
|PCC: patient-centered care|
|PCM: partial credit model|
|RMSEA: root mean square error of approximation|
|SC-eHEALS: simplified Chinese eHealth literacy scale|
|SES: socioeconomic status|
|SRMR: standardized root mean square residual|
|TIC: test information curve|
|TLI: Tucker-Lewis index|
|VAS: visual analogue scale|
Edited by G Eysenbach, R Kukafka; submitted 11.03.20; peer-reviewed by S Mitsutake, E Neter, Y Chu, M Fiordelli; comments to author 09.09.20; revised version received 25.09.20; accepted 15.11.20; published 07.12.20
©Richard Huan Xu, Lingming Zhou, Sabrina Yujun Lu, Eliza Laiyi Wong, Jinghui Chang, Dong Wang. Originally published in the Journal of Medical Internet Research (http://www.jmir.org), 07.12.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.