Measuring eHealth Literacy in the European Economic Area, Switzerland, and the United Kingdom: Scoping Review

Introduction

Rationale

The internet is used by many individuals in the context of health. In a 2024 survey conducted in the European Union, 58% of the participants aged between 16 and 74 years reported that they had used the internet to search for health-related information within the last 3 months [1]. The internet offers more than access to health-related information. Today, a variety of digital tools offer further use cases in the context of health. For example, users can generate health information via generative artificial intelligence (GenAI)–based chatbots [2] and share health information or interact with others via social media platforms [3]. In addition, dedicated health applications such as Apple Health [4] and Google Fit [5] help individuals track health goals via smartphones and smart watches. The World Health Organization provides an overview of digital interventions for individuals in the context of health that enable communication, tracking, reporting, financial transactions, and consent management [6].

However, digital tools are also associated with obstacles for individuals who lack access to hardware and internet or digital literacy [7]. Subgroups, such as older adults, are particularly at risk of digital exclusion [8,9]. Furthermore, users are confronted with misinformation on health topics. A review of studies that analyzed health-related posts on social media platforms such as Facebook, Instagram, and Twitter, now called X, indicates that health misinformation is widespread [10]. Similarly, studies show that GenAI models generate unreliable health information in some cases [11,12]. Users who follow incorrect advice generated by ChatGPT may be exposed to serious health risks [12].

As often argued in the literature, sufficient eHealth literacy (eHL) is required for the reliable use of digital tools in the context of health [13-16]. Norman and Skinner [13] proposed an early and influential definition of eHL in 2006, long before the GenAI era. They defined eHL as “the ability to seek, find, understand, and appraise health information from electronic sources and apply the knowledge gained to addressing or solving a health problem” ([13], p. 1). Information on eHL levels is relevant for providers of digital interventions to develop targeted solutions or physicians to suggest appropriate digital health interventions and to evaluate the level of support required by individual patients [17].

Different instruments have been developed to measure eHL and used to varying degrees in empirical research. The most common eHL measurement instrument is the eHealth Literacy Scale (eHEALS) [18-20]. The eHEALS is an 8-item self-report measure developed by Norman and Skinner [17] and published in 2006. It was initially tested among students aged 13 to 21 years [17] and has been used in a variety of studies, involving internet users and the general population. However, researchers pointed out limitations of the eHEALS, including its one-dimensionality [21,22], insufficient capture of actual eHL levels due to the risk of overestimation and underestimation [14,23], and lack of items related to shifting use cases, such as social media [14,24]. Researchers also argued that the underlying Lily model, named after its visual arrangement of 6 literacies around eHL—traditional literacy, information literacy, media literacy, computer literacy, scientific literacy, and health literacy [13,17]—does not cover important aspects of eHL, such as the ability to formulate health questions; know-how in using digital technologies; and contextual, cultural, and communicative competencies [15] or individual, situational, and environmental factors [25]. Another aspect missing in the Lily model is mentioned in the eHealth Literacy Framework (eHLF) by Norgaard et al ([26], p. 533): “safety and control,” particularly related to personal health data. In light of the conceptual discourse and shifting use cases, further eHL measurement instruments have been developed. Prominent examples are the multidimensional eHealth Literacy Questionnaire (eHLQ) [27], which is based on the eHLF [26], and the Digital Health Literacy Instrument (DHLI), which covers skills related to social media [14]. Similar to the eHEALS [17], these instruments provide eHL scores based on self-reports [14,27]. However, the DHLI is at least supplemented by a set of performance-based items [14]. A review indicates that performance-based approaches are less common in empirical research [18]. In March 2025, Norman, Skinner, and colleagues [28] published an updated definition of eHL, which takes “effectiveness, safety and helpfulness” into account and is as follows: “the ability to engage with digital technologies in effective, safe, and helpful ways to achieve health goals” ([28], p. 1). They also updated the Lily model, which now considers contextual factors surrounding eHL, and developed a revised 10-item version of the eHEALS based on the new Lily model, which has not yet been published [28].

Overall, the outlined historical development shows that the definition and operationalization of eHL has changed since the initial definition by Norman and Skinner [13] was published in 2006. Measuring eHL in a fast-paced digital environment is a challenging task. Researchers and health care providers not only need suitable eHL measurement instruments but also need to select appropriate data collection methods and modes, especially if they aim to assess the eHL of vulnerable subgroups. For example, the use of online surveys can lead to an underrepresentation of non-internet users [29]. To provide guidance for researchers, practitioners, and policymakers, there is a need for an up-to-date overview of current eHL measurement instruments, data collection methods and modes, and associated limitations. The latter is particularly important, as existing reviews in the field of eHL measurement [eg, 18-20] did not systematically analyze eHL measurement–related limitations reported in the included papers.

Objectives

This paper has two central objectives: (1) to provide an overview of eHL measurement instruments as well as data collection methods and modes recently used in empirical research in the European Economic Area, Switzerland, and the United Kingdom (ie, in countries that are well comparable in terms of health care system standards, culture, socioeconomic aspects, and data protection) and (2) to examine the limitations of the eHL measurement instruments and data collection approaches reported in the included papers. In addition, it aims to analyze the overall publication trend, eHL assessment frequency per participant, countries of data collection, geographical distribution of eHL measurement instruments, data collection languages, target groups, and sample sizes.

The following research question served as the basis for this paper: how has eHL been assessed in the European Economic Area, Switzerland, and the United Kingdom in recent years, and which methodological limitations need to be considered?

Methods

Given the objective of this paper, to provide a broad overview of empirical papers in the field of eHL measurement and associated limitations, a scoping review was conducted. The scoping review is reported according to the PRISMA-ScR (Preferred Reporting Items for Systematic reviews and Meta-Analyses Extension for Scoping Reviews) (Checklist 1)[30].

Protocol and Registration

The study protocol was published on PROSPERO on February 07, 2025 (CRD42025642890). It was updated on November 06, 2025, as the scope and methods—including the eligibility criteria to reduce the number of relevant records—were refined, and again on March 05, 2026, following the updated searches conducted on January 28, 2026. As can be seen in the protocol, the study was initially planned as a rapid review but evolved into a scoping review, which maps existing eHL measurement approaches and associated limitations.

Eligibility Criteria

Multimedia Appendix 1 provides an overview of the inclusion and exclusion criteria developed by TSB and JVW in consultation with JPE and JN. Papers that described empirical studies and were published in peer-reviewed academic journals were included. Papers had to be published in 2020 or later for two reasons: (1) rapid technological advances and associated new use cases in the context of health, for example, in the field of GenAI [2] and (2) regulatory changes. For example, Germany introduced the Digital Care Act (Digitale-Versorgung-Gesetz [DVG]) in 2019, which has regulated and enabled the reimbursement of prescribable digital health applications (Digitale Gesundheitsanwendungen [DiGA]) [31]. Regulatory measures regarding the regulation and reimbursement of digital health interventions have been taken in other European countries as well [32,33]. In addition, papers were included only if abstracts and full texts were available in either English or German. Papers had to describe studies with a clear geographic focus on the European Economic Area, Switzerland, and the United Kingdom. These countries are well comparable in terms of culture and socioeconomic aspects and have developed health care systems. Furthermore, data protection is strictly regulated: in the European Economic Area by the General Data Protection Regulation (GDPR) [34], in Switzerland by the Federal Act on Data Protection (FADP) [35], and in the United Kingdom by the UK GDPR and the Data Protection Act 2018 [36]. Papers had to report the use of all items or subscales of an original eHL measurement instrument intended for calculating eHL scores.

Papers were excluded if they were reviews. Study protocols, conference papers, gray literature, and book chapters were also excluded. Moreover, cross-country studies without a clear focus on the target regions were excluded. In addition, papers were excluded if they reported the use of a measurement instrument not capturing the skills described in Norman and Skinner’s [13] 2006 definition of eHL, the use of a measurement instrument solely for eHL subdimensions(eg, traditional literacy, health literacy, information literacy, scientific literacy, media literacy, and computer literacy [13]), or the use of a context-specific eHL measurement instrument (eg, COVID-19 or health conditions). Furthermore, studies that had a clear focus on adolescents or children but not on adults were excluded. Studies involving individuals with a professional perspective (eg, health professionals, IT specialists, teachers, and trainers, as well as students in health, IT, education, or communication study programs) were also excluded given the focus of this paper on eHL measurement instruments that are suitable for the general population, patients, parents, or informal caregivers.

It should be noted that it was not initially planned to exclude papers without a specific geographic focus on the European Economic Area, Switzerland, and the United Kingdom. However, after the title-abstract screening of the records from the initial searches, many records from a variety of countries with differences in culture, socioeconomic conditions, data protection regulation, and health care quality remained. To improve the comparability and facilitate the synthesis, the geographic focus eligibility criterion was added. The search strategy itself would not have changed if this criterion had been defined from the beginning.

Search Process

The reporting of the search process adheres to the PRISMA-S (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Literature Search Extension) checklist [37].

Information Sources

To identify a wide range of potentially relevant records from different subject areas, the databases PubMed and CINAHL were separately searched (ie, not simultaneously on a single platform). Supplementary searches were conducted via Google Scholar. Study registries, online or print sources, and cited references or citing references were not searched, browsed, or examined. No additional studies were retrieved through contacts or others, and no additional information sources or search methods were used.

Search Strategy

All authors collaboratively developed the search strategy. Broad search strings in English and German were used to identify a wide range of possibly relevant records via CINAHL and PubMed. Shorter search strings were used for the supplementary search via Google Scholar, as the search engine does not process long search queries [38]. The search strings were developed on the basis of the population, concept, and context (PCC) framework [39]. As the aim was to identify empirical papers that described studies involving individuals without a professional perspective (population), no subgroup-specific terms were included. Instead, the search strings contained terms and available MeSH terms in CINAHL and PubMed related to measurement instruments (concept) and eHL (context). The initial searches were conducted on January 31, 2025. In these initial searches, records published since 2020 were filtered. The searches were updated on January 28, 2026, to ensure that the review was up to date. The searches were identical, with the exception of the filters for the publication period, which were set to the period starting January 31, 2025 (PubMed), and January 2025 (CINAHL and Google Scholar), to avoid overlap with records that had already been identified in the initial searches. During both the initial and updated searches, filters were applied in CINAHL and PubMed to identify only English and German records with an available abstract. The “[a]pply equivalent subjects” expander was used in CINAHL to identify additional relevant records. Due to the large number of results retrieved in Google Scholar, the “[s]orted by relevance” function was applied, and the first 100 results retrieved via the English search string and the first 100 retrieved via the German search string were included at both search dates (ie, 400 records in total). Multimedia Appendix 2 presents the full search strategies for the CINAHL, PubMed, and Google Scholar (ie, search strings, filters, and expanders). The search strategies were not peer-reviewed.

Selection of Sources of Evidence

The screening of the records identified via the initial searches and updated searches was carried out separately at 2 different points in time. The method was identical except for one difference. The geographical focus eligibility criterion was applied only during the full-text screening of the records identified via the initial searches and not during the title-abstract screening. During the screening of the records identified via the updated searches, this criterion was applied from the beginning.

In both cases, EndNote was used for automatic deduplication. Further duplicates were later removed manually. ASReview, an open-source machine learning tool, which ranks records based on the estimated relevance, was seen as a solution to handle the high number of records identified [40]. Van de Schoot et al ([40], p. 125) showed that ASReview “can yield far more efficient reviewing than manual reviewing while providing high quality.” The records were divided by language because ASReview may have difficulty processing different languages: (1) English titles and abstracts and (2) German titles or abstracts. As most records were in English, only records with English titles and abstracts were screened in ASReview, while the smaller number of records with German titles or abstracts were screened manually.

To train the ASReview model, an initial training set containing relevant and irrelevant records was required [40]. JVW imported all English records into Rayyan, a digital tool for collaborative screening [41], created a 5% sample using the sampling function, screened the titles and abstracts and identified relevant records: 33 from the sample of the initial searches, and 2 from the sample of the updated searches. To reduce potential bias in the training set, the same number of irrelevant records was randomly selected. TSB and JN reviewed and confirmed both sets of relevant and irrelevant records. The English records were imported into ASReview, and the training set was used to train the ASReview model. ASReview default settings were used (ie, feature extraction technique: term frequency-inverse document frequency (TF-IDF), classifier: naïve Bayes, query strategy: maximum, and balance strategy: dynamic resampling/double). JVW conducted the title-abstract screening in ASReview. As recommended, the records were screened until a predefined stopping criterion was reached [40]. In line with previous review processes [42-44], the ASReview stopping criterion was set at 100 consecutive irrelevant records. The remaining English records were excluded.

Records with German titles or abstracts identified via both searches were manually screened in Rayyan. In both cases, a 25% sample was selected using the sampling function. TSB and JVW independently screened the records in both samples. Conflicts were resolved through discussion between the two authors. JVW screened the remaining records.

English and German full-text reports were retrieved and imported into Rayyan. In both cases, a 25% sample was created using the sampling function; TSB and JVW independently screened these records. Most conflicts were resolved through discussion between both authors. Some conflicts were discussed in the whole research team. JVW screened the remaining full-text reports and consulted TSB in cases of uncertainty to ensure a consistent and accurate selection.

Data Charting Process

All authors developed a standardized extraction form, tested it in advance, and modified it on the basis of a sample of 5 included papers [39] and reviewer feedback. JVW extracted data from the included papers using MAXQDA (version 24; VERBI Software Consult Sozialforschung GmbH) to manually code relevant text passages according to predefined categories. TSB then verified the correctness of all extracted data.

Data Items

The final extraction form contained the following deductive categories: (1) author and publication year, (2) title, (3) full-text language, (4) main objective, (5) objective to validate an eHL measurement instrument, (6) data collection period, (7) eHL measurement frequency per participant, (8) (target) age groups, (9) specific participants’ perspectives, (10) country focus, (11) data collection language, (12) sample size in the central eHL measurement part(s), (13) eHL measurement instrument, (14) data collection method and mode, and (15) reported limitations regarding the eHL measurement instrument and data collection method and modes. Additional subcategories were developed inductively based on the material. Information that was not explicitly reported in the included papers was inferred from the context where reasonable, as indicated in Multimedia Appendix 3.

Synthesis of Results

The synthesis focused on the following eight themes and questions:

1. Publication trend: how did the number of publications develop?
2. eHL measurement instruments: which eHL measurement instruments were used, and which versions were validated in the included papers?
3. Data collection methods and modes: which data collection methods and modes were used?
4. Reported limitations: which limitations regarding eHL measurement instruments and data collection methods and modes were reported?
5. eHL measurement frequency per participant: was eHL measured once or at multiple time points?
6. Countries and languages: in which countries and languages were data collected, how often were the eHL measurement instruments used per country, and in which languages were data collected?
7. Target groups: what were the target groups in terms of health-related perspective and age?
8. Sample sizes: how large were the samples in the central eHL measurement part(s)?

As the primary objective of this paper was to analyze how eHL was measured, the role of eHL as a variable in each included paper was not analyzed. Because the reporting was sometimes insufficiently transparent and traceable, it was not always possible to determine whether the included papers analyzed the same datasets as other included papers. Therefore, to ensure a consistent and objective approach, each included paper was considered in the synthesis, regardless of whether analyses were based on the same datasets.

Results

Selection of Sources of Evidence

Through database searches, 13,657 records were identified (CINAHL: n=3525; PubMed: n=10,132). Including the 400 records from Google Scholar, the total number of records was 14,057. Of these, 10,256 records were identified during the initial searches (January 31, 2025) and 3801 records during the updated searches (January 28, 2026). Of the records from the initial searches, 1944 duplicates were automatically removed using EndNote, resulting in 8312 unique records. Of the records from the updated searches, 509 duplicates were removed, resulting in 3292 unique records. Therefore, the total number of records to be potentially screened was 11,604. While all unique German records from the initial searches (113/113, 100%) and the updated searches (100/100, 100%) were manually screened in Rayyan, 2523 of the 8199 unique English records from the initial searches (30.77%) and 1115 of the 3192 unique English records from the updated searches (34.93%) were screened in ASReview until the predefined stopping criterion was reached. In total, 3076 irrelevant records and 7753 that remained after the ASReview stopping criterion was reached were excluded. The full texts of the remaining 775 records were retrieved and assessed for eligibility. For various reasons, 643 full-text reports were excluded. A total of 132 papers were included in the final analysis. Figure 1 shows the combined results of both selection points in time (initial and updated searches).

Some papers might appear to be eligible for inclusion but were excluded. For instance, some papers [eg, 45-48] reported only the use of selected subscales or items. As inclusion in the final analysis of this study required the reported use of all items or subscales of a version of an eHL measurement instrument intended for calculating eHL scores, these papers were excluded. Other examples of excluded papers include those that reported the use of context-specific versions of eHL measurement instruments (eg, related to COVID-19 [eg, 49-54] or chronic pain [55]).

‎

Characteristics and Results of Sources of Evidence

Multimedia Appendix 3 presents the metadata extracted from the included papers [22,58-188].

Synthesis of Results

Publication Trend

The included papers were published between 2020 and 2025, with the average number of publications per year being 22 (SD 8.72). As shown in Figure 2, the number of publications per year showed an overall upward trend. Data collection took place between 2016 and 2025.

‎

eHealth Literacy Measurement Approaches and Associated Limitations

Measurement Instruments

The included papers reported the use of all items or subscales intended for calculating eHL scores of versions of a total of 9 measurement instruments in various languages:

• the eHealth Literacy Scale (eHEALS),
• the eHealth Literacy Questionnaire (eHLQ),
• the digital health literacy module of the Health Literacy Survey 2019‐2021 (HLS₁₉-DIGI),
• the Readiness and Enablement Index for Health Technology (READHY),
• the Digital Health Literacy Instrument (DHLI),
• the eHealth Literacy and Use Scale (eHLUS),
• the revised eHealth Literacy Scale-Extended (eHEALS-E),
• the eHealth Literacy Scale for Carers of People with Chronic Diseases (eHEALS-Carer), and
• the Transactional eHealth Literacy Instrument (TeHLI)

The eHEALS was by far the most common eHL measurement instrument (94/132, 71.21%). The eHLQ was a distant second (20/132, 15.15%), followed by the HLS₁₉-DIGI (9/132, 6.82%) and the READHY (7/132, 5.30%). Of the included papers, 2 each reported the use of the DHLI (2/132, 1.52%), the eHLUS (2/132, 1.52%), and the revised eHEALS-E (2/132, 1.52%). One paper each reported the use of the eHEALS-Carer (1/132, 0.76%) and the TeHLI (1/132, 0.76%). Six of the 132 included papers4.54%) reported the use of all items or subscales intended for calculating eHL scores of 2 of the identified eHL measurement instruments: the eHEALS and the eHLUS (2/132, 1.52%) [91,166], the eHEALS and the eHLQ (2/132, 1.52%) [97,98], the eHEALS and the revised eHEALS-E (1/132, 0.76%) [86], and the eHEALS and the TeHLI (1/132, 0.76%) [160]. Table 1 lists the original eHL measurement instruments, including the initial developer and the year of development and the underlying concept or instrument.

Multimedia Appendix 4 summarizes the characteristics of the identified original eHL measurement instruments. Notably, all identified measurement instruments provide eHL scores based on self-reports. The DHLI is the only identified eHL measurement instrument, which contains supplementary performance-based items [14]. Both included papers reporting the use of the DHLI (2/132, 1.52%) did not report the use of these items [154,180].

Of the 132 included papers, 23 (17.42%) reported the objective to validate an eHL measurement instrument. These papers were grouped into two clusters:

1. Two of the 132 included papers (1.52%) reported the objective to validate a new eHL measurement instrument: the eHLUS (1/132, 0.76%) [166] and the revised eHEALS-E (1/132, 0.76%) [22]. Although an earlier version of the eHEALS-E was published in 2017 [21], the revised version validated in the included paper [22] is considered a new eHL measurement instrument in this scoping review because it contains different items.
2. Twenty-one of the 132 included papers (15.91%) reported the objective to validate a version of at least 1 already existing eHL measurement. These were versions of the eHEALS (15/132, 11.36%) [62,71-73,75,86,92,93,104,122-124,131,181,182], the eHLQ (4/132, 3.03%) [96,97,142,158], the revised eHEALS-E (1/132, 0.76%) [86], and the TeHLI (1/132, 0.76%) [160]. One included paper (1/132, 0.76%) reported the objective to validate such a version of 2 of the identified instruments: the eHEALS and the revised eHEALS-E [86]. In addition, 1 included paper (1/132, 0.76%) aimed to validate the HLS₁₉-DIGI in various languages across 13 countries [117].

Multimedia Appendix 5 provides an overview of characteristics of the validation papers and the eHL measurement instruments.

Some translated or adapted versions used or validated in the included papers differ from the original instruments in key aspects, such as dimensionality (eg, versions of the eHEALS showed a 2-factor [62,73,75,86,92,123,194] or 3-factor structure [71] unlike the original eHEALS [17]) and number of items (eg, the Greek version of the revised eHEALS-E [86] comprises 30 items 86 from the initial item pool of the revised eHEALS-E [22], and the Greek version of the eHEALS [86] showed a better fit with 6 items). Further examples of deviations include the reported use of the eHEALS with a different response scale—for example, the reported use of a “7-point Likert scale” ([139], p. 9) instead of the original 5-point Likert scale [17]—and 2 supplementary eHEALS items for calculating eHL scores [105,134].

Data Collection Methods and Modes

All included papers (132/132, 100%) reported that data on eHL were collected via surveys. As shown in Table 2, most included papers (71/132, 53.79%) reported or implied that surveys were administered digitally (eg, online or on a tablet). Twenty-three of the 132 included papers (17.42%) reported or implied that 2 or 3 data collection modes were used. One of these (1/132, 0.76%) described a cross-country study in which different modes (1 or 2) were used depending on the country [117].

Reported Limitations

Clusters of Reported Limitations

By reviewing the Limitations sections or, if these were not available, the Discussion sections, it was identified that 65 of the 132 included papers (49.24%) reported at least 1 limitation related to the applied eHL measurement instruments and eHL-related data collection methods and modes [22,58,59,62,66,67,72,73,75-79,81,82,86,87,92,97,98,104,106-109,111,112,114-117,120,122-125,128-131,134,135,137,143,144,146,152,156-159,163,166,167,170-174,180,181,183,185-187]. Four clusters were identified: (1) self-report biases, (2) selection biases due to the data collection modes, (3) instrument-specific limitations, and (4) other limitations associated with the data collection methods and modes.

Self-Report Biases

Notably, 37 of the 132 included papers (28.03%) reported the risk of self-report biases associated with the use of self-report instruments [22,58,59,62,72,73,75-77,92,97,104,106,108,109,111,112,114,123-125,130,131,134,135,137,146,152,157,159,167,171-174,181,185]. Of these, some explicitly mentioned particular forms or causes, including overestimation or underestimation of actual skills (8/132, 6.06%) [73,77,111,157,159,171,172,181], inaccurate or socially desirable responses (6/132, 4.54%) [22,77,111,131,134,171], recall difficulties (3/132, 2.27%) [77,135,137], comprehension issues (1/132, 0.76%) [75], and perspective shifts (1/132, 0.76%) [104].

Selection Biases Due to the Data Collection Modes

Twenty-six of the 132 included papers (19.70%) reported potential selection biases due to the data collection modes [62,66,67,79,81,82,92,97,98,109,115-117,120,123,124,129,130,143,146,156,167,173,183,186,187]. Most of these (22/132, 16.67%) noted that data collection via online surveys potentially led to an overrepresentation or underrepresentation of subgroups with specific characteristics and provided specific examples of potentially underrerepresented subgroups [62,79,81,82,92,97,98,109,115-117,120,123,124,129,130,143,156,167,173,186,187], including low digital literacy, less digital experience, or lack of access to hardware or the internet (19/132, 14.39%) [62,79,81,82,92,97,98,109,115,116,120,123,124,129,130,143,167,173,187]; older age (6/132, 5.54%) [79,82,109,130,173,187]; low education or functional literacy (4/132, 3.03%) [130,156,173,186]; low eHL (2/132, 1.52%) [117,167]; low income or wealth (2/132, 1.52%) [186,187]; and residence in rural areas (1/132, 0.76%) [186] (if characteristics were reported only for potentially overrepresented subgroups, these were inverted). Two of the 132 included papers (1.52%) addressed the possibility of selection biases associated with the use of self-administered, paper-based questionnaires, mentioning the exclusion of individuals with a lower level of functional literacy [66,67]. One of the 132 included papers (0.76%) noted the possible overrepresentation or underrepresentation of individuals with low eHL “even though we used paper questionnaires” ([146], p. 682). One of the 132 included papers (0.76%) reported unspecified selection biases because of fewer responses to paper-based questionnaires than to online questionnaires [183].

Instrument-Specific Limitations

Eighteen of the 132 included papers (13.64%) reported at least 1 limitation specific to an eHL measurement instrument; however, limitations regarding the validation process of translated and adapted instruments were not analyzed in this scoping review. Eleven of the 132 papers (8.33%) reported eHEALS-specific limitations, including limited comprehensiveness in times of evolving digital use cases (818/132, 6.06%) [73,86,87,120,122,125,143,167] in areas such as “Health 2.0” ([73], p. 8), “Web 2.0” ([122], p. 15), “Web 2.0 and 3.0” ([86], p. 14; [87], p. 9), and “online communities, social media, and AI-based health applications” ([87], p. 9); insufficient coverage of skills related to evaluation of digital health information (1/132, 0.76%) [107]; insufficient explanation of the connection between the eHEALS and the underlying Lily model (1/132, 0.76%) [75]; lack of clarity whether “insufficient health literacy, digital literacy, or a combination hereof” leads to low eHL (1/132, 0.76%) ([75], p. 246); insufficient distinction between individuals with different levels of digital literacy (1/132, 0.76%) [75]; lack of “uniformly used cutoff scores reported in the literature” (1/132, 0.76%) ([128], p. 11); the potential “inability to capture true differences between participants achieving the highest possible score” due to a ceiling effect found in the study (1/132, 0.76%) ([167], p. 9); and the difficulty for participants to perceive “differences between the individual items” ([128], p. 11). Three of the 132 included papers (2.27%) reported eHLQ-specific limitations: the complexity in analyses due to the multidimensional structure (2/132, 1.52%) [163,173], “increasing the risk of serendipitous results” (1/132, 0.76%) ([163], p. 10), and the less extensive psychometric testing compared to the eHEALS (1/132, 0.76%) [146]. Two of the included papers (1.52%) mentioned READHY-specific limitations: one of these (1/132, 0.76%), which was published in 2023 and reported the use of an English version of the READHY, addressed the overall complexity and length, potentially leading to selection bias and limiting its practicality [144], whereas the other (1/132, 0.76%), which was published in 2020, mentioned a lack of validated translations into languages other than Danish at the time of the survey [170]. Two of the included papers (2/132, 1.52%) that reported the objective to validate a version of the revised eHEALS-E mentioned the insufficient distinction between 2 dimensions (ie, awareness of sources and recognition of quality and meaning) [22,86]. Further criticism included the following: potential presence of obsolete items and overall length, which could impair the practicality (1/132, 0.76%) [22]; reliance of the items in one dimension (“Being smart on the Net”) on the technological progress, potentially requiring ongoing updates (1/132, 0.76%) [22]; the need to integrate further items concerning “Web 3.0” (1/132, 0.76%) ([86], p. 13); lack of cognitive interviews for some Slovenian items and all English items aimed at improving the wording (1/132, 0.76%) [22]; the availability of validated versions in Slovenian and Greek only at the time of the publication of the included paper (1/132, 0.76%) [86]; and the requirement of further testing of items to avoid social desirability bias (1/132, 0.76%) [22]. One paper (1/132, 0.76%) that aimed to initially validate the eHLUS reported that it is based on eHEALS and pointed out limitations of the development and validation process: potentially limited diversity of experts who participated in the development and validation process; operational challenges during the expert interviews; and the context-specific validation, which may not be generally applicable [166].

Other Limitations Associated With the Data Collection Methods and Modes

Other limitations identified to be associated with the data collection methods and modes included smaller sample sizes due to the exclusive use of paper-based surveys (2/132, 1.52%) [158,159], less thoughtful responses due to the use of a telephone-based survey (1/132, 0.76%) [78], and unspecified “obvious inherent limitations” of surveys (1/132, 0.76%) ([180], p. 7). In addition, one included paper (1/132, 0.76%) reported the use of different modes to collect data in different countries, limiting comparability of the results between countries [117].

Other Findings

eHealth Literacy Measurement Frequency Per Participant

Most included papers (121/132, 91.67%) reported eHL results from a single measurement per participant [22,58-67,69,70,72,74-102,104-130,132-159,161,162,164,165,167-175,177-180,183-188]. Seven of the 132 included papers (5.30%) aiming to validate a version of an eHL measurement instrument used a test-retest design [71,73,131,160,166,181,182]. Three of the 132 included papers (2.27%) reported eHL results from baseline eHL measurement and 1 follow-up (1/132, 0.76%) [68] or 2 follow-ups (2/132, 1.52%) [103,176], and 1 included paper (1/132, 0.76%) reported only the results from 2 follow-ups due to a technical error [163].

Countries and Languages

Countries

The described studies were conducted across 22 countries within the target regions of this study, most frequently in Germany (39/132, 29.54%). Five included papers (5/132, 3.79%) reported multicountry data collection, with 1 paper each reporting data collection in 2 countries (1/132, 0.76%) [85], 4 countries (1/132, 0.76%) [143], 6 countries and additional unspecified countries within the European Union or the European Economic Area (1/132, 0.76%) [59], 7 countries (1/132, 0.76%) [58], and 12 countries within the target regions of this study and 1 country (Israel) located outside the target region of this study (1/132, 0.76%) [117]. Table 3 lists the countries of data collection identified in the included papers.

Geographical Distribution of eHealth Literacy Measurement Instruments

The use of the eHL measurement instruments varied considerably by country. Versions of the eHEALS were used across 19 different countries within the target regions of this study. In contrast, the other identified eHL measurement instruments were used much less frequently. Apart from the eHEALS, only the eHLQ, the HLS₁₉-DIGI, and the READHY were each used in more than 2 countries. The distribution of the eHL measurement instruments by country of data collection in the included papers is shown in Figure 3.

‎

Languages

As shown in Table 4, the described studies collected data within the target regions of this paper in a total of 22 languages, most frequently in German (41/132, 31.06%). Two of the 132 included papers (1.52%) reported data collection exclusively in a language other than the official language of the country of data collection (Sweden), namely Arabic [66,182]. Ten of the 132 included papers (7.58%) described studies that collected data in 2 or more languages, including 1 paper (1/132, 0.76%) that reported data collection in both the country’s official language (Spanish) and a regional official language (Catalan) [97], 2 papers (2/132, 1.52%) that described multicountry studies in which data were collected exclusively in the official language(s) of the respective countries within the target regions of this study [117,143] (one of which also reported data collection in nonofficial languages in Israel [117], which lies outside the predefined target regions of this study), and 7 papers (7/132, 5.30%) that reported data collection not only in the official language(s) of the respective countries within the target regions of this study but also in other languages—English (4/132, 3.03%) [58,59,153,179]; Arabic (1/132, 0.76%) [65]; Russian and Turkish (1/132, 0.76%) [152]; and Arabic, English, Russian, Turkish, and Twi (1/132, 0.76%) [128].

Samples

Health-Related Perspectives

Three major clusters were identified regarding the health-related perspectives of the target groups:

1. Sixty-four of the 132 included papers (48.48%) reported that patients or individuals with current or previous health conditions were the target group [61-64,70,71,74,75,87-89,91,93,95,96,99,100,103,106,108,110,111,114-116,118,120-122,124-127,129,130,136,144-149,153,156-159,161-165,167-171,174,177,179,180,183-185].
2. Eleven of the 132 included papers (8.33%) stated that parents or legal guardians of children or informal caregivers were the target group [85,101,104,112,113,132,133,135,158,172,176].
3. Fifty-nine of the 132 included papers (44.70%) described studies with other target groups (eg, general population or students) [22,58-60,65-69,72,73,76-84,86,90,92,94,97,98,102,105,107,109,117,119,123,128,131,134,137-143,150-152,154,155,160,166,173,175,178,181,182,184,186-188].

Two of the 132 included papers (1.52%) were each assigned to 2 of these clusters. Of these, one study (1/132, 0.76%) reported that patients (assigned to cluster 1) and parents of hospitalized children (assigned to cluster 2) were the target group [158], and the other (1/132, 0.76%) reported that users of symptom checker applications (assigned to cluster 1) and nonusers of symptom checker applications (assigned to cluster 3) were the target group [184]. One of the included papers (1/132, 0.76%) reported that, in addition to patients with hypertension (assigned to cluster 1), physicians were the target group [70]; no target group cluster 4 for physicians was created, as this study did not focus on individuals with a professional perspective.

Age Groups

In terms of age groups, the included papers were grouped into three clusters:

1. Of the 132 included papers, 119 (90.15%) reported that only adults were included. Of these papers, 104 (104/132, 78.79%) stated that adults in general aged ≥18 or >18 years or broad adult age ranges were included [22,58,59,61,62,65-67,69-71,75,77-79,81-90,92,93,95-101,103,105-108,110-121,123-131,133,136-139,142-153,155-160,162-167,170-174,177,178,180-185,187,188]. Furthermore, 10 of the 132 papers (7.58%) focused on older adults, namely adults aged ≥40 years (2/132, 1.52%) [102,186], >45 years (1/132, 0.76%) [63], ≥50 years (1/132, 0.76%) [68], 50 to 69 years (1/132, 0.76%) [161], ≥60 years (1/132, 0.76%) [74], and ≥65 years (4/132, 3.03%) [64,94,122,169]. Two papers (2/132, 1.52%) mentioned that younger adults were included, namely adults aged 18 to 29 years (1/132, 0.76%) [175] and 18 to 35 years (1/132, 0.76%) [80]; 2 papers (2/132, 1.52%) reported that middle-aged adults were included, namely adults aged 30 to 65 years (1/132, 0.76%) [168] and 34 to 64 years (1/132, 0.76%) [91]; and 1 paper (1/132, 0.76%) reported that both “younger adults” and “older adults” were included, without further specification ([134], p. 1739).
2. Five of the 132 included papers (3.79%) reported that minor and adult participants were included. Of these, 1 paper (1/132, 0.76%) mentioned that individuals with a verified social media account were the target group, regardless of their age, with the actual age range being 14 to 72 years [72]; 3 papers (3/132, 2.27%) specified that individuals aged ≥14 years (1/132, 0.76%) [76] and ≥16 years (2/132, 1.52%) [109,154] were included; and 1 paper (1/132, 0.76%) stated that individuals aged <35 years were included [73].
3. Eight of the 132 included papers (6.06%) did not provide a target age, a qualitative description regarding the age, an actual age range, or age groups [60,104,132,135,140,141,176,179].

Sample Sizes

As shown in Table 5, the included papers were divided into 7 clusters on the basis of the sample sizes reported for the central eHL measurement(s), or total sample if not available, ranging from <50 (7/132, 5.30%) to >2000 participants (8/132, 6.06%).

Discussion

This paper aimed to provide an overview of how eHL has been assessed in the European Economic Area, Switzerland, and the United Kingdom in recent years and which methodological limitations need to be considered.

Principal Findings

Summary of Evidence

A total of 132 empirical papers were included in the final analysis. The publication trend indicated growing research activity in the field of eHL measurement. Across the included papers, the use of versions of 9 eHL measurement instruments that provide eHL scores based on self-reports was reported, most frequently the use of the eHEALS (94/132, 71.21%). All included papers (132/132, 100%) reported the use of surveys to collect data on eHL, most of which were conducted digitally (71/132, 53.79%). Reported limitations include potential self-report biases associated with the use of self-report instruments (37/132, 28.03%), selection biases due to the data collection mode (26/132, 19.70%), and limitations specific to 5 of the 9 identified eHL measurement instruments (18/132, 13.64%). Most included papers (121/132, 91.67%) reported eHL results from a single measurement per participant. Data were collected across 22 countries within the target regions, most frequently in Germany (39/132, 29.54%). The use of the eHL measurement instruments varied considerably between the countries. In total, 22 different data collection languages were identified, with German being the most common (41/132, 31.06%). Patients or individuals with current or previous health problems were a commonly reported target group (64/132, 48.48%). Broad adult age ranges were very common (104/132, 78.79%). Samples sizes ranged from ≥50 (7/132, 5.30%) to >2000 (8/132, 6.06%) participants.

Self-Report eHealth Literacy Measurement Instruments

Self-report biases are a key limitation of self-report measurement instruments [198,199], as also noted in various papers included in this scoping review. A study, which was also cited in some included papers, found a weak relationship between the eHEALS—the most frequently used eHL measurement instrument—and performance-based task [14,23]. However, no performance-based eHL measurement instruments were identified via this scoping review. One reason may be that we did not consider instruments solely assessing only some of the skills described in the 2006 definition of eHL by Norman and Skinner [13] or focusing on specific subdimensions of eHL. Crocker et al ([18], p. 1) identified “29 unique performance-based eHealth literacy measurement tools,” using less strict criteria. One of these was used in a study also included in this scoping review: Schulz et al [155] reported the use of the eHEALS alongside additional tasks. Specifically, participants rated the quality of 2 simulated websites (1 with high quality and 1 with low quality) using a 7-point semantic differential scale (eg, “accurate” vs “inaccurate”) ([155], p. 4); furthermore, they were shown the right treatment options (eg, “seeking help from a doctor”) ([155], p. 4]) and wrong treatment options (eg, “treating depression with St John’s wort or vitamins or yoga without any mention of antidepressant medications or psychotherapy”) ([155], p. 4]) and had to make a selection. While Crocker et al [18] considered this approach an eHL measurement instrument, it does not capture the relevant aspects of the eHL definition by Norman and Skinner [13] (eg, searching health information) and was therefore not considered an eHL measurement instrument in this paper. Another instrument, identified by Crocker et al [18] and via this scoping review, is the DHLI [14]. It comprises 21 self-report items intended for calculating eHL scores and 7 supplementary performance-based items, which did not demonstrate acceptable validity in the validation study [14]. Thus, the DHLI [14] is a self-report eHL measurement instrument with supplementary performance-based items rather than a performance-based eHL measurement instrument. Nevertheless, Crocker et al [18] found that performance-based eHL measurement instruments are rarely used and highlighted that several were not completely published. They also mention other challenges of performance-based eHL measurement instruments, such as the requirement of regular updates considering new scientific evidence, varying topic relevance for different target groups, and the time demand for researchers and participants [18].

Exclusion of Subgroups in Online Surveys

Selection biases were mentioned in the included papers, particularly in the context of frequent use of online surveys. Online surveys potentially introduce self-selection biases and lead to an undercoverage of subgroups, such as individuals with limited or no access to the internet [200]. Despite growing internet adoption, 6% of households in the European Union still had no internet access in 2024 [201]. Hardware and internet access, in particular, are prerequisites for using digital tools [202]. However, even individuals who do not yet use the internet may benefit from digital health interventions if they are equipped with the necessary hardware and an internet connection. In addition, digital interventions need to reflect eHL-related needs [203]. Therefore, it would be important to ensure inclusion of subgroups who are not yet using the internet, and other potentially underrepresented subgroups, which were mentioned in the included papers. Selection biases can be addressed through methodological strategies, such as probability sampling [200]. Another approach is mixed-mode designs (ie, different modes are offered to different target groups)—which, however, introduces further (selection) biases [204]—as well as sequential mixed-mode designs (ie, different survey modes are offered to the same target group one after another) [205].

Alignment With Technological Innovations

This study found that some included papers criticized eHL measurement instruments for not fully aligning with technological developments and shifting use cases. As the internet has been getting more interactive since the eHEALS was published in 2006, skills related to social media, where users potentially encounter health misinformation [10], are considered relevant in the context of eHL [23,24]. Although social media is not a new phenomenon, only some identified eHL measurement instruments contain items explicitly related to interactive digital tools: the revised eHEALS-E [22], the TeHLI [196], and the DHLI [14]. Also, the HLS₁₉-DIGI [189], which is based on the DHLI [14], contains such items, although not in the main part intended for calculating eHL scores (HL-DIGI). eHL can also be considered essential in the context of health information provided by GenAI, as studies [11,12] show that health misinformation is a serious issue. Yet, GenAI has been established in recent years, and this may explain why it is not explicitly mentioned in older eHL measurement instruments. However, not even the revised eHEALS-E [22] and the eHLUS [166] published in the period after the launch of ChatGPT in 2022 mention GenAI tools explicitly. However, the developers in the revised eHEALS-E took GenAI into account, pointing out “that the Being smart on the Net dimension is particularly important in the context of the rapidly evolving internet technologies based on artificial intelligence algorithms that require critical awareness, distance, and understanding” ([22], p. 8). Dedicated measurement instruments were developed in recent years to measure artificial intelligence literacy in general [206,207]. Furthermore, as outlined, newer theoretical models such as the eHLF [26] and the new Lily model [28] also include the aspect “safety.” However, safety in terms of data protection is only considered in the DHLI [14], the eHLF-based eHLQ [27], and the eHLUS [166]. Overall, the findings indicate the need to continuously review and update eHL measurement instruments to ensure alignment with technological innovations. Developing an eHL measurement instrument that does not become obsolete in a rapidly changing digital environment can be described as a key challenge for future research.

Complexity of eHealth Literacy Measurement Instruments

In terms of complexity or length, the eHEALS-E [22], the eHLQ [27], and the READHY [190], which contains the eHLQ [27], were criticized in the included papers. Similar to the DHLI [14], the TeHLI [196], and the eHLUS [166], these eHL measurement instruments capture a broader range of dimensions of eHL. Efthymiou et al [86], who validated the Greek versions of the eHEALS and the revised eHEALS-E, provided guidance in selecting the appropriate eHL measurement instrument. Specifically, they noted that “[t]he eHeals could be used as an additional tool when eHealth Literacy is not the core concept measured and the revised eHeals-Extended can be used when researchers wish to measure eHealth Literacy concept more thoroughly” ([86], p. 2). Following this argumentation, the use of eHL measurement instruments with a broader range of dimensions should be considered when the aim is to generate deeper insights, including specific needs of individuals. However, it is debatable whether health professionals will follow this advice. For example, Norman ([24], p. 2) noted that the eHEALS was developed as a short scale, as “health professionals […] said they would not use a long instrument in practice.”

eHealth Literacy Measurement Instruments for Older Adults

In addition, we want to highlight that none of the eHL measurement instruments identified in this paper was specifically developed for older adults. Also, Wang et al [208] noted a lack of such eHL measurement instruments. The researchers recently introduced the Digital Health Literacy Questionnaire for Older Adults “designed to align with the realities of older adults’ daily lives and health care experiences, focusing on potential challenges they may face when using digital technologies to access, process, communicate, and understand health information and services“ ([208], p. 12). The instrument is not included in the final analysis of this scoping review, as it was used to assess eHL of older adults in China [208]. Older age groups differ from younger individuals in certain characteristics (eg, physical and cognitive conditions, literacy levels, technology usage, and privacy preferences), requiring special methodological considerations, for example, in terms of item complexity [209]. As older adults are particularly at risk of digital exclusion [9,210], there is a need to evaluate the appropriateness of eHL measurement instruments for them.

Limitations

The review process used is subject to some limitations. First, it needs to be noted that, due to the use of ASReview to facilitate the title and abstract screening of the English records, 30.77% (2523/8199) of the English records from the initial searches and 34.93% (1115/3192) of the English records from the updated searches were screened. The remaining English records remained after the ASReview stopping criterion (ie, 100 consecutive irrelevant records) was reached in both cases. Thus, despite the demonstrated performance of ASReview [40], there is a risk that relevant papers were not screened and thus not included in the final analysis.

Second, the exclusion of papers without a clear focus on the target regions resulted in the omission of papers and eHL measurement instruments. As German and English papers were included, there may be an overrepresentation of papers from German- and English-speaking regions and an underrepresentation of papers from non–German-speaking and non–English-speaking regions within the target regions.

Third, the analysis covered limitations reported in the Limitations section or, if a Limitations section was unavailable, the Discussion section of the included papers. Therefore, limitations reported elsewhere in the included papers were not analyzed. Moreover, the analysis was limited to reported limitations regarding eHL measurement instruments as well as eHL-related data collection methods and modes. Other limitations reported in the included papers (eg, those related to cross-sectional study designs or sampling or recruitment strategies and limitations specific to translated and culturally adapted versions of eHL measurement instruments) were not extracted and analyzed in this study.

Implications for Practice, Future Research, and Policy

This paper provides health care practitioners and researchers with a structured overview of limitations related to eHL measurement instruments and data collection methods and modes. The identified eHL measurement instruments differ in key aspects, including length, complexity, and alignment with rapidly evolving digital use cases. Particularly, online surveys might lead to an exclusion of subgroups. Thus, it clearly shows that selecting both a suitable eHL measurement instrument and suitable data collection methods and modes is a key decision, which should be based on factors such as the purpose of the planned eHL measurement, the target population, and the data collection setting.

The overview could encourage researchers to adopt eHL measurements that are less common or were not used in specific countries yet. This potentially requires translation, cultural adaptation, and validation efforts. Future research should also focus on revising existing eHL measurement instruments and developing new eHL measurement instruments to capture the skills required for an ever-growing range of digital tools in the context of health, including social media and GenAI. This may require new underlying operationalization approaches that manage to avoid becoming obsolete. Use case–specific eHL measurement instruments may also be relevant for health care providers and researchers who only want to assess specific aspects (eg, skills that are necessary when using a specific digital health application) considering setting and time constraints (eg, in health care facilities). The fact that only self-report eHL measurement instruments were identified and the associated risk of self-report biases reported in the included papers indicate a need for performance-based eHL measurement approaches. Furthermore, revised or new eHL measurement instruments tailored to older adults are highly relevant for gaining a better understanding of the specific eHL-related needs.

Policymakers should encourage health care providers to conduct eHL measurements using carefully selected eHL measurement instruments and data collection methods and modes to provide targeted solutions based on individual eHL-related needs.

Conclusions

To our knowledge, this is the first scoping review synthesizing eHL measurement limitations reported in empirical papers from the European Economic Area, Switzerland, and the United Kingdom. The analysis of 132 empirical papers showed that eHL was assessed using 9 eHL measurement instruments in various languages, especially the eHEALS, and that digital surveys were a common data collection approach. The identified eHL measurement–related limitations, such as self-reported eHL measurement, exclusion of subgroups, and lack of alignment of technological developments and changing use cases in the context of health, potentially lead to biased results. Therefore, selecting suitable eHL measurement approaches is a critical decision. Health care professionals and researchers should consider a variety of factors, such as measurement purpose, target population, and data collection setting. There is a need for translated and culturally adapted as well as revised or newly developed eHL measurement instruments that keep pace with technological change, cover diverse use cases or focus on specific use cases, and contain performance-based components. Policymakers should encourage health care providers to thoughtfully select appropriate eHL measurement approaches and use eHL-related insights as a basis for targeted solutions.