Currently, the internet is the main source of health information. According to the survey Flash Eurobarometer 404 [1], 8 (80%) in 10 Europeans used the internet for private matters, with the majority (59%) using it for health information searching purposes. The increasing number of internet users has been influencing health systems globally, leading to investments in the provision of eHealth services. In recent years, there have been great efforts to use eHealth interventions to engage patients in health care and help them manage their health, particularly among older adults with chronic disease self-management and self-care [2-4]. Nevertheless, despite the broad access to the internet and other electronic sources, there continues to be a lack of skills to access valid and reliable health information, as well as to understand, evaluate, and apply it adequately for decision-making purposes [5-7]. Considering this condition, eHealth literacy has emerged as a new concept in health care and a challenge in public health [8].

According to Norman and Skinner [9], eHealth literacy is “the ability to seek, find, understand, and evaluate information from electronic sources and apply it to knowledge gained to address or solve a health problem.” Based on this, individuals should be able to perform basic or advanced information retrieval, distinguish documents from reliable sources (eg, authoritative ones or scientific evidence-based), and understand eHealth terminology. Furthermore, this set of skills requires the ability to use information and communication technology (ICT), think critically about their nature, and efficiently navigate different electronic resources to obtain information for health-related decision-making.

Considering that eHealth literacy is not a static set of skills, instead changing over time, Norman and Skinner [9] consider the importance of assessing and monitoring eHealth literacy as critical to supporting eHealth interventions. In recent years, indeed, older adults have become increasingly active on the internet and interested in digital health care services to manage their health; however, limited attention has been given to measuring eHealth literacy in this population. Therefore, we sought to address this gap by translating, culturally adapting, and validating a scale for Portuguese older adults that measure eHealth literacy levels (Multimedia Appendix 1).

The eHealth Literacy Scale (eHEALS), developed by Norman and Skinner [10], was the most frequently used instrument to measure eHealth literacy worldwide, covering 18 languages, 26 countries, and diverse populations and contexts (ie, adolescents, adults, elderly, patients, healthy people, caregivers, health school professionals, community, and clinical practice) [11]. Since its development, the eHEALS has been widely translated, adapted, and validated in several countries, such as Brazil [12], China [13-16], Ethiopia [17], Germany [18,19], Greece [20], Hungary [21], Indonesia [22], Iran [23,24], Italy [25,26], the Netherlands [27]; Norway [28,29], Poland [30,31], Portugal [32], Serbia [33], South Korea [34-36], Spain [37], and Sweden [38], as well as in countries with English as the main language [39-47].

The scale’s development study [10] assessed the eHealth literacy of Canadian adolescents (n=664) aged between 13 and 21 years from 14 secondary schools. Considering the psychometrics test results, the instrument showed a good internal consistency (α=.88), with moderate to strong correlations between items (0.51≤r≤0.76) and a moderate test-retest reliability (0.40≤r≤0.68). In addition, a unidimensional structure was found in exploratory and confirmatory factor analyses, explaining 56% of the total variance.

In Portugal, and similarly to the original study, the eHEALS was translated, adapted, and validated in 2014 in a sample of adolescents (n=1215) attending secondary education. The results showed a good internal consistency (α=.84) and a 2-factor structure in the exploratory factor analysis (EFA; extraction of 2 factors with α=.81 and α=.73), explaining 61% of the total variance. From a subgroup analysis, the authors found statistically significant differences regarding the level of education. For the variables sex and age, no statistically significant differences were found. No further translation, adaptation, and validation studies of eHEALS considering other target populations or contexts in Portugal have been carried out since then [32].

Studies with samples including older adults [24,36,39,43,44,48] showed good internal consistency, with Cronbach α ranging between 0.87 and 0.99. Regarding factor analysis, 3 studies [24,36,39] obtained 1D structures, and 2 other studies obtained 3D ones [43,44].

The questionnaire itself consists of 8 items assessing the person’s perception of knowledge, comfort, and ability to find, evaluate, and apply health ICT. Each of the items is scored at 5 points on a Likert scale ranging between “1” (totally disagree) and “5” (totally agree), with a total score from 8 to 40 points. The higher the score, the higher the eHealth literacy levels. Items 1 and 2 are related to awareness, items 3 and 4 are related to demand, items 6 and 7 are related to the evaluation of health resources, and items 5 and 8 are related to the use of health information. The instrument also contains 2 additional items, not adding to the final score, which assesses the participant’s perception of using the internet to access health information and make health decisions in terms of its usefulness and importance. The theoretical basis of the eHEALS was the social cognitive theory of Albert Bandura and the Lily Model, which explains multiple components of the constructs based on 6 components of literacies—traditional (literacy and numeracy skills), health, information, scientific, media, and computer [10].

This paper presents and discusses the results of full psychometric testing of the eHEALS-PT24 (Portuguese version of the eHealth Literacy Scale for older adults), which aimed to:

We followed a methodology for translating, adapting, and validating instruments proposed by Sousa and Rojjanasrirat [49]—a 7-step guideline based on the review of existing recommendations in the scientific literature on the process of translation, cultural adaptation, and validation of instruments for use in cross-cultural health care (Figure 1).

The original version of the questionnaire was independently translated to Portuguese by 2 translators with high proficiency in English: a translator with knowledge of the terminology used in health care and another with knowledge of the cultural and linguistic characteristics of the target language (forward translation; step 1). After this, a third translator compared the two translated versions of the instrument and the original one to detect discrepancies. In addition, a first meeting with a committee of experts with different backgrounds and expertise was held to evaluate discrepancies, composed of the 2 bilingual experts from the first step, the third bilingual translator, academia members, and the research team (step 2). These 2 steps generated the preliminary translated version of the instrument for the Portuguese language.

Independent, blinded backward translation to English was carried out by 2 translators whose first language was English as spoken in the United Kingdom: one translator with knowledge of the terminology used in health care and another with knowledge of the cultural and linguistic characteristics of the target language (step 3). This step generated 2 retroverted versions of the original version of the instrument, which were then compared with the original one (in English) to detect discrepancies by the research team. In addition, a second meeting with a committee of experts with different backgrounds and expertise was held to detect discrepancies, composed of all translators involved in the previous three steps, academia members, and the research team (step 4). These 2 steps generated the prefinal version of the translated instrument, which the original authors validated.

The prefinal version of the instrument was then tested in a pilot study in the target language with a monolingual sample (n=15) to evaluate the instructions, items, and response format clarity (ie, to establish whether the instrument could be satisfactorily understood and completed by people from the target population; step 5). As part of this step, a committee of experts (n=8) was also held to further examine the instrument for clarity of the instructions, items, and response format (content equivalence assessment). The fifth step generated some adjustments in the prefinal version of the instrument.

According to the protocol followed, step 6 (preliminary psychometric testing of the prefinal version of the translated instrument with a bilingual sample) is rarely used, except when a bilingual population is accessible, which justified the option of not carrying it out in this validation process.

Full psychometric testing (step 7) involved reviewing and refining the items of the final version of the eHEALS for Portuguese older adults, as well as establishing stability, internal consistency, validity, and model fit. This step encompassed 2 substudies: general psychometric characteristics of the instrument (n=80, study 1) and confirmatory factor analysis (CFA; n=301, study 2). The sample sizes were defined according to the protocol, namely, at least 10 participants per item of the instrument for general psychometric analysis and 300 to 500 participants for CFA.

Study 1 included 80 older adults from a Health Family Unit (Primary Care) located in Faro. The sampling procedures were based on a convenience sample. Data collection was conducted by in-person questionnaires between May and July 2022. During that time, patients aged 65 years or older who went to the Health Family Unit (Primary Care) for an appointment were invited to participate in the survey.

Study 2 included 301 older adults from 2 distinct Health Family Units (Primary Care), located in Faro and Lisbon. Participants were randomly selected from the total number of patients of the 2 considered Health Family Units and invited by phone to visit the unit and participate in the survey. Data collection was conducted by in-person questionnaires between May and July 2023.

Inclusion criteria were age (aged 65 years or older) and sex (female and male). Difficulty communicating or using the Portuguese language was an exclusion criterion. The questionnaire removal was performed when the questionnaires were not entirely fulfilled.

The data analysis was conducted in several stages to ensure a comprehensive understanding of the psychometric properties of the eHEALS-PT24. The following steps outline our analytical approach.

The ethics review assessments were submitted and approved by Portugal’s National Data Protection Authority and Regional Health Administrations (Algarve, Lisbon, and Tagus Valley; (approval 4825/2018). Informed consent was applied to participants, according to the model recommended by the Ethics Committee of the Algarve Regional Health Administration, guaranteeing voluntary participation, the possibility of withdrawing at any time, data anonymity, privacy, confidentiality, and no harm to participants. In addition, an authorization request was made to the original authors of the eHEALS [10], which was accepted.

Before the full psychometric testing, we carried out a pilot study, which we briefly summarized to support the results presented in this subsection. The sampling procedure for piloting was based on snowball sampling, composed of older people aged 65 years or older (n=15). Data collection was conducted using a web-based questionnaire between July and August 2020. The interrater agreement among the sample was above 80% for all items. As part of this step, a committee of experts (n=8) was also held to further examine the instrument for clarity of the instructions, items, and response format; also, to assess the content equivalence, content validity index at the item level (I-CVI=1) and at the scale level (S-CVI=1), as well as the Fleiss κ coefficient of agreement (κ=0.24; P<.001), were calculated.

The Bartlett sphericity test corroborated the factorability of the correlation matrix for both studies (study 1: χ²₂₈=446.87, P<.001; study 2: χ²₂₈=3932.81, P<.001). In addition, the KMO test value demonstrated adequate sampling for both studies (study 1: KMO=0.87; study 2: KMO=0.94). Given the quality of Bartlett and KMO values, the criteria for factor analysis were gathered. By performing the EFA, we considered factors with eigenvalues above 1 for testing the structure of the instrument. In accordance with the original structure of the eHEALS [10], the Jamovi software extracted 1 factor for the structure of the eHEALS-PT24 for both studies (Study 1 and 2).

Regarding CFA for study 2 (n=301), a single-factor model demonstrated high item loadings (standardized estimates 0.88 to 0.97; Table 6); however, the model fit was mixed: the chi-square test was significant (χ²₂₀=265, P<.001), CFI was acceptable (0.94), while TLI (0.91) was marginally acceptable, and RMSEA (0.20) indicated a poor fit. Given the RMSEA values for one dimension, and due to the variability of factor structures in the literature, we compared the indexes for 1-, 2-, and 3-factor structures in study 2 (Table 7).

For the 2-factor model, we followed Tomás et al [32], dividing items 1-5 (factor 1) and items 6-8 (factor 2). High item loadings (standardized estimates 0.90 to 0.97) were obtained (Table 6); however, the model fit was mixed: CFI was acceptable (0.96), TLI (0.94) was marginally acceptable, RMSEA (0.17) indicated a poor fit, and the chi-square test was significant (χ²₁₉=188, P<.001; Table 7).

For the 3-factor model, we followed Stellefson et al [43], dividing items 1-2 (factor 1), items 3-5 (factor 2), and items 6-8 (factor 3). High item loadings (standardized estimates 0.90 to 0.98) were obtained (Table 6); however, the model fit was mixed: CFI was acceptable (0.96), TLI (0.93) was marginally acceptable, RMSEA (0.18) indicated a poor fit, and the chi-square test was significant (χ²₁₇=179, P<.001; Table 7).

We used the mean total and SD values to determine high or low eHealth literacy levels. Therefore, participants were divided into 2 groups for each study as follows:

In study 1 (n=80), 35 (43.8%) participants had low eHealth literacy levels, and 9 (11.3%) had high levels. In study 2 (n=301), 115 (38.2%) participants had low eHealth literacy levels, and 71 (23.6%) had high levels.

We performed known-groups validity in study 2, as presented in the following subsections.

In this paper, we have presented the translation, cultural adaptation, and validation of the eHEALS for Portuguese older adults (eHEALS-PT24) by carrying out a general psychometric analysis in a sample of 80 participants (study 1) and a CFA including 301 participants (study 2). For this discussion, we highlight the findings obtained in study 2 since it represents the final stage of the validation of eHEALS-PT24, according to the followed protocol [49].

Based on scale and items stability and reliability analysis, we concluded that the eHEALS-PT24 is a reliable tool for measuring eHealth literacy. Correlations between all items of the eHEALS-PT24, ranging between moderate and strong ones, with statistical significance, indicated that the questions were related to the instrument with no redundancy. In addition, the instrument showed good internal consistency. All items were considered reliable to assess eHealth literacy levels in older adults, with high correlations between each other. Compared with the original validation study [10], internal consistency was higher (Cronbach α=0.88 vs 0.98), which is comparable with previous studies among older adults [24,36,39,43,44,48]. The only study showing a higher Cronbach α than ours was a validation in older Hispanic participants [48] (Cronbach α=0.99), but the sample was small (n=20), which was pointed out as a limitation by the authors.

Concerning construct validity, the eHEALS was originally developed with a single-factor structure [10], and in our study, the same structure was yielded in EFA, explaining 86.4% of the total variance. This total variance explained was higher compared with the original validation study [10] (56%), as well as to previous studies among older adults [24,36,39,43,44,48], suggesting the model captured a significant portion of the data’s variation. Regarding CFA, a single-factor model demonstrated high item loadings; however, the model fit was mixed: the chi-square test was significant (P<.001), CFI (0.94) and TLI (0.91) were both acceptable, and RMSEA (0.20) indicated a poor fit. The finding of a unidimensional structure also aligns with previous studies carried out in older adults [24,36,39]; however, it contradicts the results of the 2 other studies conducted among older adults, which indicated a better fit for the 3-factor structure [43,44]. The variability of factor structures in the literature and the RMSEA value obtained led us to compare the indices (ie, chi-square, CFI, TLI, and RMSEA) for 1-, 2-, and 3-factor structures. All indexes fitted well, but RMSEA remained poor (ie, 0.20 for 1 factor, 0.17 for 2 factors, and 0.18 for 3 factors), suggesting that the model may not fit the data well. These findings are in line with the eHEALS Korean validation study (CFI=0.95; TLI=0.92; and RMSEA=0.12) [36], which followed a unidimensional structure. Considering the original validation study [10], as well as others who used Rasch analysis (or other analyses under modern test theory) [25,40,41], we also decided on the 1-factor structure.

The eHEALS-PT24 mean score was 23.31, with 38.2% (115/301) of participants showing low eHealth literacy levels and 23.6% (71/301) showing high levels. Compared with other validation studies among older adults, the mean score was higher than in Spain (22.35) [48] but lower than in China (30.94) [39] and in the United States (29.05) [43]. As shown by previous studies [12,15,18,30,32], item 8 of the eHEALS-PT24 had the lowest average among all scale items, indicating that people do not feel so confident in using information from the internet to make health-related decisions compared with other eHealth literacy skills.

To further examine construct validity, known-groups validity was assessed, showing statistically significant differences in demographic variables (age, residence area, and education levels), except for sex. While a previous study among older adults [48] found no differences between eHealth literacy levels and age, we concluded that the greater the age, the smaller the eHealth literacy. Those aged 65-79 years had, on average, 7.0 points higher (eHEALS scores) than those aged ≥80 years. Regarding eHealth literacy levels and sex, Aponte and Nokes [48] pointed out significant differences, where mean values of eHEALS scores for men (13.85) were significantly lower than those in women (25.77). In our study, female respondents had, on average, 0.65 points higher eHEALS scores than male respondents; however, no statistically significant differences were found between eHealth literacy levels and sex (P=.31). We also tested the association of eHealth literacy levels and the residence area, finding statistically significant differences (P<.001), where participants living in urban areas had on average 7.08 points higher (eHEALS scores) than those living in rural areas. Regarding education levels, the highest mean values corresponded to doctoral or equivalent level (ISCED 8; 31.57), master’s degree or equivalent (ISCED 7; 29.17), and postsecondary but not tertiary education (ISCED 4; 27.68). Since some education-level cases had nonrepresentative samples (n<30), this result should be read with caution. Therefore, we can only infer participants with primary education (ISCED 1) had significant differences in eHealth literacy levels from the participants with other education levels, except for lower secondary education (ISCED 2) and master’s degree or equivalent (ISCED 7).

There are some limitations in this study. First, there was limited geographic representation since the sample did not include older people from all regions of the country (only Faro and Lisbon). If we consider, however, the sample representativeness, we believe it is suitable for providing us with an adequate snapshot of our population of interest. Second, since the web-based questionnaires were only applied in the pilot study, we did not have any rigorous confirmatory process to determine that participants were aged, in fact, aged 65 years or older. Third, similarly to other validation studies [25,27,30-32,43,44,48], test-retest reliability was not conducted. As such, further research could address reliability and responsiveness over time. Fourth, although a more comprehensive psychometric analysis could be conducted to establish the instrument validity, such as investigating correlations with external measures of eHealth literacy or health outcomes, the results of this study are promising and show that the eHEALS-PT24 is a reliable tool for perceived measurement of eHealth literacy. Fifth, the eHEALS instrument measures a narrow scope of eHealth literacy and may not fully capture the complex concept of eHealth literacy nowadays since it was developed before the social media era (Web 2.0). Nevertheless, the benefits of eHEALS (strong psychometric properties, brevity, and usability) outweigh its shortcomings, and the eHEALS arguably serves as a convenient instrument for health professionals.

In conclusion, the eHEALS-PT24 is a reliable and valid instrument for assessing and monitoring the eHealth literacy of Portuguese older adults. This instrument can be useful in identifying older adults who are able to use eHealth resources and participate in eHealth interventions aimed at engaging them in health care and helping them to manage their health and, by extension, assess the effects of eHealth interventions.