The COVID-19 pandemic has accelerated the development of digital health infrastructures, devices, and services. Different digital health solutions were used in different settings worldwide with different approaches and purposes regarding, for example, the dissemination of public information for prevention, the understanding of the disease, the generation of diagnostic tools, or the sharing of real-time data for surveillance and control of the spread of the infection [1]. This scenario provided an opportunity to develop and test the capacity of digital technologies to increase the quality of health services, which required a similar improvement in citizens’ digital skills. This is reflected in the World Health Organization global strategy on digital health 2020-2025 [2], one of whose 4 strategic objectives is to advocate for people-centered digital health systems. To this end, it is necessary to promote adequate health literacy with regard to digital health services among users and health care professionals to encourage the adoption and active use of these services.

In line with the World Health Organization strategy, and as other countries such as Ireland, New Zealand, the United Kingdom, and the United States had previously done [3-6], the Spanish Ministry of Health established, as part of its Digital Strategy 2021-2026, that one of the strategic objectives would be the empowerment and involvement of people in their health care and disease control by facilitating the relationship with health services [7]. Within this framework, people’s trust in the health system is established as the key element for the implementation of a digitalized and efficient health system [8,9].

In general terms, data from recent reports and surveys show that Spain can be considered one of the leading countries in the use of digital technologies related to health. This is also the case in Catalonia, a region in northeastern Spain with almost 10 million Catalan speakers [10]. According to Eurostat [11], almost 7 out of 10 Spanish citizens used the internet to seek health information in 2021 (European Union average 56%), which is close to the leading countries in the area (Finland: 80%; the Netherlands: 77%; Norway: 77%; Denmark: 75%). In fact, digital clinical appointments were already available in 40% of Spain in 2019 (51.7% in Catalonia), in second position behind Finland (53%). In addition, in 2021, digital medical records and electronic prescriptions were widely implemented (with different levels of access and services depending on the region). A European survey on digital health across Europe [8] showed that Spain outperforms the average of the countries surveyed in indicators such as digital clinical consultations, the use of mobile health apps, access to electronic medical records, and remote supervision for monitoring or sharing symptoms with health professionals, with only 25% of Spanish people not using any digital technology to manage their health in the last year (a proportion that rises to 36% in the rest of the world) [9]. However, the use of wearables for health monitoring together with digital access for mental health in Spain show lower indicators than those of the rest of the countries studied. These promising data are complemented by the positive attitude of Spanish citizens toward the adoption of digital health even among those who have not yet used it [8].

The development of digital health services must be accompanied by digital health literacy, skills, confidence, and a positive attitude toward the relevance of these services [12-15] so that they can respond to people’s needs in an equitable and inclusive way. Understanding these needs has been difficult due to the lack of a rigorous theoretical framework for measuring digital health literacy [16]. In 2006, Norman and Skinner [12] pioneered the development of their eHealth Literacy Scale (eHEALS) to assess individuals’ self-perceived ability to seek, find, understand, and evaluate health information from electronic sources and apply the knowledge gained to address a problem. Currently, there are different instruments available to measure digital health literacy [17]. However, the only tools currently available in Spanish are the eHEALS questionnaire [12] and the European Health Literacy Survey Questionnaire (HLS-EU-Q) [18], an instrument developed by the European Health Literacy Consortium [19]. However, the HLS-EU-Q is designed primarily to assess health literacy and not digital health literacy. The short version of the HLS-EU-Q (16 items) includes 3 items that assess the ability to evaluate reliability, protect oneself, and understand the health information available in the media (including on the internet). However, this is clearly insufficient in a digital health context. Similarly, as with the 3 items in the short version of the HLS-EU-Q, the eHEALS questionnaire does not capture the breadth of services and aspects that digital health now covers [17]. Moreover, neither instrument is available in Catalan.

To address the shortcomings of the eHEALS in moving beyond Health 1.0, the Digital Health Literacy Instrument was developed, a performance-based tool aiming to include the interactivity of the web [20]. However, it did not comprehensively capture the interactions that digital health entails, including aspects linked to individuals (information and knowledge about one’s own health), the intersection between users and technologies (the feeling of being safe and in control and their motivation), and users’ experience of the systems (that they work and are accessible and that they respond to users’ needs). To cover all these aspects, the eHealth Literacy Questionnaire (eHLQ) was developed [21,22]. The eHLQ is an instrument with 35 items distributed in 7 scales that each assess a dimension of digital health literacy (Textbox 1).

Each scale has between 4 and 6 items with 4 response options (strongly disagree, disagree, agree, and strongly agree) with an assigned value of 1 to 4. With the aim of describing the person’s relationship with digital health services in a broad way, the questionnaire does not have a total score. This is because health literacy is a relational concept [23], and it is firmly rooted in a specific context, so it depends on the characteristics of each health system. Furthermore, to make it user-friendly, the questionnaire is designed to allow for modular use (by using isolated scales) depending on the aspects that are of interest to analyze in each context (eg, people’s ability to use health services or the availability of services that respond to their needs). It is because of this descriptive intent that, rather than quantitative scores, the eHLQ analysis results in user profiles from which vignettes (scenarios) can be defined that exemplify people’s needs, skills, and competencies and provide insights into their interactions and engagement regarding their own experiences with digital health services. These can be used by policy makers and health service managers to design and offer improved services.

The eHLQ was constructed simultaneously in Danish and English to minimize cultural references [22]. It has shown good internal consistency reliability, and beyond these 2 original versions, as far as the authors of this study are aware, there are 3 more published versions available today that have shown positive evidence of psychometric quality ([24] in Mandarin [25], Swedish [26], and Dutch), although other adaptations are in progress across Europe. The eHLQ has been used to assess health literacy in different groups [27-29] both in Denmark and Australia [27,30], and further extensions of it have been developed for application in specific groups, for example, for health professionals [31] or older people, to assess their readiness to use digital health services [32]. Therefore, it is constantly evolving and a suitable tool to be used in a context of digital health growth such as that in Spain.

The main objective of this study was to undertake the translation and cultural adaptation of the eHLQ and gather evidence of psychometric quality of the eHLQ scores in a large sample of the general population in Spain, creating a simultaneous process for 2 of the official languages spoken in the country: Spanish and Catalan.

This study involved the translation and cross-cultural adaptation of the eHLQ into Spanish and Catalan. A cross-sectional design was used to gather evidence of the psychometric quality of the Spanish and Catalan versions of the eHLQ (Figure 1).

The translation and cultural adaptation process followed version 5 of the Translation Integrity Procedure [33], which provides systematic translation documentation and helps adaptation teams maximize measurement equivalence [34]. The same process was used for both the translation into Spanish and into Catalan and consisted of three main steps:

Cognitive interviews were carried out with 8 Spanish speakers and 8 Catalan speakers (12/16, 75% female; aged 22 to 81 years; 3/16, 19% with a primary school education; 5/16, 31% with a high school education; 8/16, 50% with a university education; 3/16, 19% with a rare or occasional use of technology; 6/16, 38% with a medium use of technology; and 7/16, 44% with habitual or continuous use of technology) to field-test the intelligibility and equivalence of each translation. Following a thinking-aloud approach, respondents were asked to answer each item after silently reading it. Scripted probes were used to explore difficulties and items that needed rephrasing or further explanation. Spontaneous probes were also used, for example, when the respondent hesitated or took a long time to answer an item. Changes suggested were introduced and discussed during the final consensus meeting to obtain the definitive versions of the questionnaire.

Ethics approval was obtained from the Ethics Committee of the Universitat Oberta de Catalunya (20201013_ ehernandez_Traduccio), and the license for translating the eHLQ was obtained from Swinburne University of Technology, Australia (November 11, 2020).

All participants in the validation process gave informed consent before data collection and received a gift card for digital marketplaces as compensation. Data were gathered anonymously to ensure participants’ confidentiality.

During the series of consensus meetings, which involved the forward translators, the authors of this paper, and 2 of the authors of the original versions of the eHLQ, all the items for both the Spanish and Catalan translations were examined one by one. The first meeting served to highlight the most relevant points and pressing issues, which included the following agreements, all of which were applicable to both versions: (1) “health technology systems” and “eHealth systems” could be translated together as “digital health services”; (2) the “I find that...” expression at the beginning of different items was equivalent to “I think that...” but could not be removed and substituted by a more factual and objective form as it was essential to keep the point of view of the respondent; (3) “measurements” had a broader meaning than results from medical tests as it included other data sources, such as fitness-tracking devices; (4) the “works together” expression implied really interoperable systems, not just connected systems; (5) “find” could be translated as “search” (in relation to health information searching); and (6) it was not possible to combine ≥2 words together (eg, “data/measurements”) as it was key to the questionnaire that there be no ambiguity in terms of the intent of each item.

Throughout the systematic process of translation and cultural adaptation, the issues discussed were generally the same for both the Spanish and Catalan versions. In addition to the aforementioned agreements, a number of key wording points were addressed and revised, for example, “perfectly” versus “very well,” “being certain” versus “being sure,” “available” versus “at the disposal of,” “skills” versus “capabilities,” or simply word order according to linguistic and cultural uses. A few specific issues also arose from cognitive interviews that required rephrasing both in the Spanish version (Table 1) and the Catalan version (Table 2). In almost all cases, the issues were the same. Scale 4 (Feel safe and in control) did not need any changes after the cognitive interviews in either the Spanish or Catalan version.

Cognitive interviews raised the need to reinforce the information initially included in the original English version used. As well as adding a few more examples to the existing ones for health professionals, health technology, and digital health services, further information was included after issues were raised regarding items eHLQ26, eHLQ5, and eHLQ12.

Item eHLQ26 was particularly problematic as it was hard to express everything encapsulated in the word “measurements” with just one word in Spanish or Catalan. The possibility of using more than one word was ruled out as it is key to the questionnaire that there be no ambiguity in terms of the intent of each item. Adding “data” or “results” alongside “measurements” would open the door to respondents stating that they use one but not the other. In the end, the decision was made to translate “measurements about my body” as “data on how my body works.” In the instruction page, a new specific section was included to explain what “data on how my body works” meant by providing a few examples.

Similarly, items eHLQ5 and eHLQ12 raised the need to include a new communication and sharing information section on the instructions page, reinforcing the idea that having conversations or sharing information could be both informal and with professionals.

The main objective of this study was to undertake the translation and cultural adaptation of the eHLQ and gather evidence of the psychometric quality of the eHLQ in a large stratified sample of the general population in Spain, creating a parallel process for 2 of Spain’s official languages: Spanish and Catalan.

Overall, the new versions of the eHLQ in Spanish and Catalan were found to be highly coherent with the original validation of the English and Danish versions [22]. As shown by the simultaneous process carried out for the original versions to maximize comparability over linguistic differences, the Spanish and Catalan versions present similar patterns that were confirmed through measurement invariance in 2 different language versions within the same sociocultural context.

The Translation Integrity Procedure [33] provided common ground and facilitated iterative discussions among the professional translators, the authors of the original questionnaire, and the authors of this study, leading to a final consensus that ensured that the translated versions maintained the same semantic and conceptual orientations as the original version [45]. Furthermore, this process generated some interesting insights and qualitative results regarding the conceptualization of the instrument and its equivalence in a context of different languages [34].

The cognitive interviews, which provided validity evidence based on the test content and the response process, showed that some of the items could be misunderstood in the new translated versions, and it was difficult to reach a consensus to overcome the cultural adaptation. This was particularly relevant for item eHLQ26 (I use measurements about my body...) as the term “measurements” in this setting is mainly associated with height and weight in both Spanish and Catalan. Thus, the wording of this item had to be changed slightly to “I use data on how my body works,” and some specific examples of this concept were included in the instructions page with the terms used in the questionnaire. Although the difficulties with the concept of measurements (including professional and fitness-tracking devices) can be related with the low indicator for Spain in a European survey [7], this item has also been problematic in previous studies [15,24,26]. For example, in the Australian and Taiwanese populations, this item had low factor loadings, and in the Dutch population, it had residual correlation issues. In this study, this item presented high factor loadings in both versions.

Along the same lines, other groups of examples of terms related to “communication and sharing of information” (both with professionals and other people) were also included to further clarify the intent of items eHLQ20 (I use technology to share information...) and eHLQ27 (Technology improves my communication...). In fact, although recent reports and surveys show that Spain can be considered one of the leading countries in the use of digital technologies related to health [8,10], it seems that the concepts of “eHealth,” “digital health,” and “digital health services” are not well understood by the general population. Despite the fact that citizens are increasingly using them, there is little familiarity with these terms outside professional circles, highlighting the need for an instrument such as the eHLQ to assess the readiness of the population and help increase the adoption and use of digital health services.

Reinforcing the instructions page with more information forms part of the cultural adaptation process, and the eHLQ counts on this. The Spanish and Catalan versions of the eHLQ took advantage of this possibility to add more information. However, it did not provide further recommendations on their use, administration, or interpretation, as do other versions [26].

Results for both the Spanish and Catalan versions showed means around 2.5 points, with the 7 scales showing a similar deviation pattern. The values were in a similar range to that of the values obtained in the studies in Denmark (between 2.42 and 2.97) [22], Sweden (between 2.58 and 3.04) [25], and Taiwan (between 2.37 and 3.08). In all 3 cases, scale 2 (Understanding of health concepts and language) had the highest score, and scales 7 (Digital services that suit individual needs; for the Danish and Swedish versions) and 6 (Access to digital services that work; for the Mandarin version) had the lowest mean.

The 7-factor correlated model presented adequate GOFIs both in the Spanish and Catalan versions. Moreover, measurement invariance was found between both forms for the factor loadings. The CFA 1-factor model for each scale also fit well considering CFI and TLI, and all the items presented a strong factor loading within their respective scales. In this model, the RMSEA values had good results for some of the scales in the Spanish version, although this index can be problematic for assessing models with small df, as in this study [24].

According to the factor loadings, all the items were around the constraint value of each scale. In both versions, the scale with the highest loadings was scale 4 (Feeling safe and in control), as mentioned in recent surveys in Spain as the key element for the implementation of a digitalized and efficient health system [8,9]. For the Spanish version, almost all items had factor loadings of >0.80 except for 2 items of scale 6: eHLQ9 (r=0.77; My health care providers deliver services that I can access through technology) and eHLQ16 (r=0.78; My health data are available to me wherever I am). Item eHLQ9 was also the one with the lowest factor loading in the Catalan version. This cannot be considered a low factor loading as these results are in line with those of the original version, where these items showed the lowest factor loadings (0.44 and 0.43, respectively) [22]. Ensuring access and universal coverage for digital health systems, represented by scale 6, is still a major challenge for individuals to process and understand.

There was a high correlation between the dimensions, especially for factors 1 and 5 (Using technology to process health information and Being motivated to engage with digital services), both in the Spanish and Catalan versions (r=0.90 in both cases). These values resemble the findings for the original version, which the authors attributed to the result of being located along the same causal path [22]. Both versions showed other high correlations between factors 5 and 7 (r=0.92 in Spanish and r=0.89 in Catalan), perhaps suggesting that motivation and engagement regarding using digital health services are related to the perception that those services suit the users’ specific needs and preferences.

To explore convergent and discriminant validity evidence, the eHEALS [12] was also administered, which is a good measure of some elements of digital health literacy. It has been adapted and has shown evidence of psychometric quality in Spanish in a sample of university students [36]. Mean scores were similar in the Spanish (27.35, SD 8.40) and Catalan (27.80, SD 7.08) versions and slightly lower than those in a study on the use of the internet and digital literacy among Spanish nurses, with a mean of 32.44 [46]. Moderate to low correlations of the eHEALS were found with the different eHLQ scales, as expected because of their different theoretical models [12,21]. This was an expected result given that, even though the eHEALS is still a widely used tool, it does not provide sufficient understanding of the individual’s interaction with digital services and technology or the user’s experience of engaging with the system, which are all covered by the eHLQ’s dimensions. For both translated versions of the eHLQ, the lowest correlation with the eHEALS was found for scale 4 (Feel safe and in control; r=0.56 in Spanish and r=0.43 in Catalan), probably because all the items within this scale relate to the ownership of personal data and their secure access, which are elements of a dimension not included at all in the eHEALS. Lower correlations were expected between the eHEALS and scales 3, 4, 5, 6, and 7, and discriminant evidence was searched for. The high correlation among the other subscales could be explained by the low discrimination among the subscales of the eHLQ, as was also the case in previous studies and as has been described previously.

As for internal consistency reliability, adequate values were found for all the scales even considering the minimal number of items (4-6) per scale, with Cronbach α and categorical ω values of >0.80 (and almost all >0.90) in all the scales in both the Spanish and Catalan versions. These values confirm that the selected items are consistent in each scale in both translated versions and are in line with the original results of the Danish validity testing study [22], with internal consistency values ranging from 0.77 to 0.86, and with the translation and adaptation to the Mandarin language [24], with values ranging from 0.75 to 0.95. In both studies, the lowest internal consistency values were also found for scale 2. Although the Swedish version [25] shows higher values (ranging from 0.82 to 0.92), scale 2 (together with scale 6) also shows the lowest internal consistency values.

Moreover, reliability was considered good as all the scales in both versions had α and ω values of >0.80, with values close to 0.90.

In both the Spanish and Catalan versions, all the scales showed the sensitivity to discriminate among known groups according to their educational and socioeconomic level and their self-categorization as technology users. However, results must be interpreted with caution as measurement invariance across the groups could not be studied due to sample size limitations. This is a relevant issue as a previous study in a Dutch sample could not establish full invariance among the different groups [26]. The differences were statistically significant for all the scales in these 3 known groups. Thus, people with a primary school education, lower socioeconomic status, or occasional use of technology scored lower than those with university studies, higher socioeconomic status, or continuous use of technology. This was apparent in all the scales and in both the Spanish and Catalan versions, with similar patterns of dispersion. Although the difference among the groups reveals the discrimination power of both versions of the questionnaire, this finding has to be taken with caution as it may also be an artifact due to the large samples and the dispersion shown by the scores.

Finally, a tendency to mark “disagree” rather than “strongly disagree,” especially for those groups with lower socioeconomic status, was revealed across the scales. This might reflect some respondents’ attempts to hide a lack of comprehension or the inappropriateness of the questions to their particular situation, prompting them to provide more socially desirable answers. The possibility was discussed of including some previous instructions to encourage people to provide an answer for all the items and answer “strongly disagree” in cases in which the question was not clearly understood or did not apply to their own situation. However, as answers should not be suggested to respondents under any circumstances, a general recommendation designed to encourage people to respond as well as they could was included, informing them that there were no correct or incorrect answers. In relation to that, the questionnaire raises aspects linked to digital health (eg, services and devices) that the Spanish and Catalan populations, even as educated and digital users, do not recognize in their current daily lives. Considering the rapid development of digital health in Spain, this will no longer be the case in the near future. For this reason, future testing studies of the eHLQ will probably not require such an extensive introduction to the questionnaire as the one that was included as a key element in the cultural adaptation of the instrument in this study.

The translation and cultural adaptation of an instrument into 2 languages simultaneously is a strength of this study because they provide a tool with psychometric guarantees to assess health literacy. In addition, the stratified sample of the Spanish- and Catalan-speaking populations in terms of gender, age, and place of residence is another important strength that makes the eHLQ a good instrument to use in community and population surveys.

However, all the interviews were conducted using a digitally administered questionnaire, and therefore, all the respondents needed some basic level of digital skills, which may have excluded some people with no internet access or no basic skills from the sample. To overcome this limitation, the sample was balanced by conducting 200 (n=100, 50% for each version) face-to-face interviews in community settings targeted at people with primary school studies and of a low socioeconomic level, and 80% (60/200) of them were aged >60 years. In this sense, it is also important to note that, according to the self-categorization scale regarding technology use, the total sample included respondents who reported that they never or occasionally used technology (54/400, 13.5% in the Spanish version and 27/400, 6.7% in the Catalan version).

Regarding future studies, although the sample in this study was large and covered a wide range of sociodemographic, academic, and technology use profiles, it was not free of voluntary reporting and self-reporting biases. Further studies are suggested involving participants with different health conditions to provide evidence of the psychometric quality of the instrument in different contexts of use.

Future studies with larger sample sizes should consider testing for measurement invariance across groups to test whether group differences are due to measurement issues or real differences. Finally, it would be interesting to consider the development of further versions covering Basque and Galician, which are also spoken in Spain.

This study demonstrates that the translated and adapted versions of the eHLQ in Spanish and Catalan are compatible with the original’s construct and item intent and maintain proper psychometric properties. The eHLQ provides a tool in Spain for researchers, policy makers, and health service managers to explore people’s needs, skills, and competencies and obtain interesting insights into their interactions and engagement regarding their own experiences with digital health services, especially in the context of digital health growth in Spain.