We conducted a facility-based cross-sectional study and collected data from January 21 to March 10, 2025, at Churchill Health Center in Addis Ababa and Cheleleka Health Center in Bishoftu. Addis Ababa, Ethiopia’s capital, is the urban center of the country with about 6 million people living in the metropolitan area [57]. Bishoftu is a town of about 200,000 inhabitants located 50 km southeast of Addis Ababa [58]. The Ethiopian public health system follows a 3-tiered structure, with primary hospitals, health centers, and health posts at the primary level; general hospitals at the secondary level; and specialized hospitals at the tertiary level [59]. Both study sites are primary-level health care facilities. This cross-sectional study was reported in accordance with the STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) guidelines.

For data collection, we used a questionnaire comprising the Amharic eHLQ, the Amharic eHEALS [29], items on internet and technology use, and sociodemographic information (age, gender, education, occupation, and income). Participants completed the eHLQ using a 4-point Likert scale (1=strongly agree to 4=strongly disagree) and the eHEALS using a 5-point Likert scale (1=strongly agree to 5=strongly disagree). While the eHEALS provides a global score, the eHLQ does not, and its scales are interpreted independently.

We obtained a license from Swinburne University, Australia, to translate the English eHLQ into Amharic. We applied the Swinburne Translation Integrity Procedure to ensure measurement equivalence [60]. Two native Amharic speakers fluent in English conducted the forward translation: one prepared the initial draft, and the other reviewed it. After discussing discrepancies, the version was jointly reviewed with a member of the eHLQ development team. A native Amharic speaker with high English proficiency, who had not seen the original questions, conducted the back translation. The final consensus meeting, chaired by a member of the eHLQ development team, involved all translators, bilingual team members from Ethiopia, and team leaders from Ethiopia and Germany. The group reviewed all items until agreement was reached. Four items (6, 9, 23, and 26) required detailed discussion because of translation challenges. For item 25, which is about using technology to organize health information, the local team raised concerns about contextual appropriateness, as organizing health information is typically conducted by health professionals, not patients. No item was removed or significantly modified to maintain the scale’s psychometric integrity.

One trained data collector conducted cognitive interviews with 9 Amharic speakers in Addis Ababa to identify potential challenges in item interpretation. The sample aimed to represent demographic variety and included patients, office workers, and a nurse from Churchill Health Center in Addis Ababa, as well as older adults personally known to the data collector. Ages ranged from 29 to 62 years, and educational levels ranged from finishing 10th grade to master’s degree. We did not specify internet access as an inclusion criterion for the cognitive interviews, as we initially intended to validate the questionnaire in the general population. All participants provided written informed consent, and the interviews were audio-recorded with permission. Participant feedback was collected using the think-aloud method [61] and analyzed using inductive coding.

Challenges emerged across items of all scales, mainly related to translation clarity, technical terminology, contextual relevance, and conceptual interpretation. Translation issues: Item 23 was unclear because the translation of “work together” implied active collaboration. Technical terminology: Participants, particularly individuals with limited digital exposure, found terms such as “digital health technologies” difficult to interpret consistently (eg, items 7, 8, and 10). Contextual relevance and applicability: Limited familiarity with technology and digital health technologies in particular caused challenges (eg, items 8, 9, 18, and 23). Conceptual interpretation: Some participants struggled with vague references (eg, “people who are supposed to”) in items 1 and 3. Item 21 was often interpreted as referring to external body functions (eg, movement of the hand) rather than internal functions. Item 26 raised uncertainty regarding whether “measurements” referred only to digital tools or included manual methods. Furthermore, a few items were perceived as repetitive, and older participants reported greater difficulty completing the questionnaire.

Based on the pretest findings, adjustments were implemented. We revised the translation of “work together” in item 23 to better reflect the idea of system coordination in the context of different technologies. Furthermore, we trained data collectors to use the eHLQ introductory information, thereby minimizing interpretation challenges and clarifying item intent when needed. Finally, we refined the inclusion criteria, adding smartphone ownership and access to the internet, to ensure that participants had access to technology and the internet to enable meaningful engagement with the questionnaire.

We recruited 2 experienced Ethiopian data collectors with backgrounds in public health and medical care and provided a 1-day training, which included a comprehensive review of the questionnaire and a practical session to pilot data collection procedures. Data were collected in a REDCap database [62].

We recruited participants using convenience sampling at Churchill Health Center in Addis Ababa and Cheleleka Health Center in Bishoftu. Inclusion criteria included having experience as a patient in the Ethiopian health care system, being 18 years of age or older, the ability to read a text in the Amharic language, ownership of a smartphone, and access to the internet. Smartphone ownership and internet access were chosen as the eHLQ measures users’ competencies and perceptions related to digital health services. Informed by the cognitive interviews, these criteria were intended to support meaningful item interpretation and reduce construct-irrelevant variance related to lack of digital access [63]. Churchill Health Center in Addis Ababa and Cheleleka Health Center in Bishoftu were selected as recruitment sites because both are primary health care facilities in urban settings where Amharic is widely spoken and access to the internet and smartphones is comparatively high [64].

Based on the recommendation by Comrey and Lee [65], we aimed to recruit at least 300 participants.

During data collection, 356 individuals were approached with support from health care staff. Of these, 54 were not eligible: 31 did not own a smartphone, 5 did not have internet access, and 18 had neither. This left 302 participants. In addition, participants with more than 50% missing data were excluded (n=2). The final analytic sample comprised 300 participants. Missing data were handled using listwise deletion for the confirmatory factor analysis (CFA) with the weighted least squares mean and variance–adjusted (WLSMV) estimator and pairwise deletion for Spearman correlations.

We conducted data analysis using R (version 4.5.0; R Foundation for Statistical Computing) [66] and Mplus (version 8.11; Muthén and Muthén) [67]. Structural equation modeling was performed with the lavaan package [68]. Descriptive statistics were used to summarize participants’ demographics. For continuous variables, we reported means (SD) or medians (IQR). For categorical variables, we presented absolute and relative frequencies. Mean scores were calculated for each eHLQ scale, with lower scores indicating stronger agreement. Floor and ceiling effects in items were defined as ≥15% of responses in the lowest or highest category [69].

Psychometric evaluation followed the Standards for Educational and Psychological Testing [63], gathering evidence from multiple sources. We examined structural, convergent, and discriminant validity as well as internal consistency. As no causal effect estimates were planned, no adjustment for potential confounders was performed. Because the Amharic eHLQ is a translation of a prestructured questionnaire, we computed a CFA using the WLSMV estimator, which is suitable for ordinal data [70,71]. Missing data were handled through listwise deletion, resulting in 251 observations being included in the CFA.

Standardized factor loadings above 0.4 were considered acceptable, following the cutoff applied in the Swedish validation study [43]. R² values were calculated to estimate the proportion of variance explained by each factor.

We assessed model fit using the following Goodness-of-Fit Indices: chi-square (χ²) [72], comparative fit index (CFI) [73], Tucker-Lewis index (TLI) [74], root mean square error of approximation (RMSEA) [75], and standardized root mean square residual (SRMR) [72]. Established cutoff values are CFI and TLI >0.95, RMSEA <0.06, and SRMR <0.08 [72,75]. Compared to other fit indices such as RMSEA and CFI, the SRMR is considered more stable across different estimation methods [76,77]. However, robust cutoffs for fit indices using the WLSMV estimator are not yet available [78,79]. Therefore, model fit was evaluated in comparison with prior CFA findings from translations of the eHLQ (Multimedia Appendix 1) and considered acceptable if within the reference ranges CFI ≥0.93, TLI ≥0.92, SRMR ≤0.09, and RMSEA ≤0.09.

Scale intercorrelations were assessed using Spearman correlations between the mean scores of the eHLQ scales, with 95% CIs obtained via bootstrapping.

Given the high interfactor correlations reported in previous studies, we explored 3 alternative model structures: a single-factor model to assess the distinctiveness of the factors and a second-order and a bifactor model to evaluate a potential higher order or general factor. Model comparisons were performed using chi-square difference tests.

Internal consistency was assessed using Cronbach α and McDonald ω values, with 95% CIs obtained via bootstrapping. Values ≥0.70 for both tests were considered acceptable [80,81].

To assess convergent and discriminant validity, we administered the Amharic eHEALS [29] as a reference instrument. The expected pattern of correlations derives from the conceptual mapping of the eHLF, which underpins the eHLQ, to the Norman and Skinner Lily model, which underpins the eHEALS [25,35,36,53]. As shown by Norgaard et al [35], the 6 literacies of the Lily model are distributed across the first 3 domains of the eHLF, which are operationalized in eHLQ Scales 1 to 3. The remaining 4 eHLF domains have no equivalent in the Norman and Skinner model and are operationalized in eHLQ Scales 4 to 7 [35]. Hence, we expected convergent validity with the eHEALS for eHLQ scales 1 to 3 and discriminant validity for eHLQ Scales 4 to 7. Total scores of eHEALS were correlated with scores from each eHLQ scale using Spearman correlation with 95% CIs. We interpreted large correlations (ρ≥0.7) for Scales 1 to 3 as evidence of convergent validity, whereas weak correlations (ρ<0.4) for Scales 4 to 7 as evidence of discriminant validity [82].

This study received ethical approval from the School of Public Health, Addis Ababa University (ref number EP&BI/274/2025) and the Medical Faculty of Martin Luther University Halle-Wittenberg, Germany (processing number 2024‐223). All participants provided written informed consent. Data were collected using REDCap [62]. Participants’ privacy and confidentiality were maintained throughout the study. All data were recorded anonymously, and no identifying information was stored. No compensation was provided for participation.

The mean age of participants was 30.4 (SD 6.8, range 18‐55) years (Table 1). Of the 300 participants, 209 (69.6%) were women, and 271 (90.3%) lived in an urban area. Regarding education, 59 (19.7%) had completed grade 12, whereas 121 (40.3%) had attained higher education. More than 9 out of 10 participants (276/300, 92%) used technology (eg, smartphones) at least once a day, and 230 (76.7%) used the internet at least once a day.

Mean scores on the eHLQ scales ranged from 2.0 (SD 0.4) on Scale 4 (“feeling safe and in control”) to 2.4 (SD 0.5) on Scale 2 (“understanding of health concepts and language”). Ceiling effects were observed in 6 items across different scales (items 1, 2, 3, 6, 7, and 14), and floor effects were found in 1 item (item 26; Table 2).

We fitted a 7-factor model to the present data (Table 3). Values of SRMR, CFI, and TLI (0.07, 0.97, and 0.97, respectively) aligned with values of prior validation studies (Multimedia Appendix 1), indicating a satisfactory model fit. The RMSEA (0.10, 90% CI 0.09‐0.10) was slightly higher than values reported in prior validation studies (Multimedia Appendix 1). In addition, 34 out of 35 items showed loadings above 0.4 (Multimedia Appendix 2). One exception was item 1 of Scale 4 (“feeling safe and in control”) (λ=.18; P=0.01; R²=0.03). Thirty out of 35 items showed loadings above 0.5. All factor loadings had P=.01 (Multimedia Appendix 2).

Scale intercorrelations ranged from 0.36 (95% CI 0.23‐0.47) between Scale 4 and Scale 7 to 0.84 (95% CI 0.79‐0.88) between Scale 3 and Scale 7 and 0.84 (95% CI 0.80‐0.88) between Scale 5 and Scale 7. Most correlations were high, with the majority being above 0.7 (Table 4).

The estimation of the bifactor model was unsuccessful due to nonconvergence and was therefore excluded from further analysis. The results of the single-factor and second-order factor models are presented in Table 3. The chi-square difference test to evaluate differences of the second-order factor model and the 7-factor model rejected the corresponding null hypothesis (χ²₁₄=411.78, P<.001), indicating that the more complex model, including 7 first-order factors and a second-order factor, explains the data best.

Internal consistency was acceptable to excellent across most scales, with Cronbach α ranging from 0.72 (95% CI 0.63‐0.77) to 0.91 (95% CI 0.89‐0.93) and McDonald ω from 0.79 (95% CI 0.72‐0.83) to 0.96 (95% CI 0.94‐0.97). Only Scale 4 (“feeling safe and in control”) showed a lower Cronbach α of 0.62 (95% CI 0.51‐0.70) but an acceptable McDonald ω of 0.70 (95% CI 0.61‐0.77). For the total score, representing the second-order factor of eHealth literacy, Cronbach α was 0.96 (95% CI 0.96‐0.97) and McDonald ω was 0.97 (95% CI 0.96‐0.98) (Table 5).

The mean of the Amharic eHEALS total scores was 20.1 (SD 6.9), with observed total scores ranging from 8 to 40. We found strong positive correlations between the Amharic eHEALS total scores and Scales 1, 2, and 3 of the eHLQ (ρ=0.70, ρ=0.69, and ρ=0.74, respectively), supporting convergent validity (Table 4). Although the correlation with Scale 2 was slightly (ρ=0.01) below the predefined cutoff of ρ=0.70, we cautiously interpreted it as evidence of convergent validity. Correlations between the Amharic eHEALS total score and Scales 4 to 7 of the Amharic eHLQ ranged from ρ=0.41 (95% CI 0.31‐0.50) for Scale 4 (“feeling safe and in control”) to ρ=0.67 (95% CI 0.60‐0.67) for Scale 5 (“ability to actively engage with digital services”) and ρ=0.67 (95% CI 0.61‐0.73) for Scale 7 (“digital services that suit individual needs”). With the correlations of Scales 5, 6, and 7 exceeding 0.6, reflecting moderate associations, our findings provide only limited evidence for discriminant validity between the Amharic eHLQ and the Amharic eHEALS (Table 4).

This study aimed to translate the eHLQ into Amharic and evaluate its psychometric properties. We assessed structural, convergent, and discriminant validity, as well as internal consistency. Our results showed satisfactory fit for a 7-factor model, robust internal consistency in line with expectations for comparatively short scales [83], acceptable evidence for convergent validity, and mixed evidence for discriminant validity.

Six items showed ceiling effects, and 1 item showed a floor effect, indicating limited differentiation at the extremes for only a few items, while the overall response distribution was adequate. In comparison, the Swedish eHLQ found ceiling effects in most items, possibly reflecting greater confidence in digital health skills and systems among the Swedish sample, which may be partly due to Sweden’s highly digitalized health care [43,49,84].

With a scale ranging from 1 (strongly agree) to 4 (strongly disagree), mean scores between 2.0 and 2.4 indicate an average to a slightly below-average level of eHealth literacy across the eHLQ scales. The highest mean score was observed for Scale 2 (“understanding of health concepts and language”) and Scale 6 (“access to digital services that work”), suggesting greater disagreement on these scales. Since Scale 2 assesses aspects of health literacy [35], this may reflect low health literacy levels, as previously reported in Ethiopia [85]. Disagreement on Scale 6 may reflect barriers to accessing technology and digital health services. For example, as described by the local research and clinical teams, patients have only limited access to their personal health data [6].

We observed the lowest mean score for Scale 4 (“feeling safe and in control”), suggesting that most participants felt safe and in control regarding data storage and authorized access. However, cognitive interview results and follow-up research team discussions suggested difficulty in the comprehensibility of Scale 4 items in the local context, which might be reflected in the comparably less favorable psychometric properties, as discussed below.

The CFA supported the 7-factor structure of the Amharic eHLQ. The SRMR, CFI, and TLI indicated an overall satisfactory model fit, as they fell within the range of values from previous validation studies (Multimedia Appendix 1). The slightly elevated RMSEA was not considered critical, particularly because the SRMR is regarded as more robust with the WLSMV estimator [76].

All factor loadings showed exploratory P=.01 (Multimedia Appendix 2). Item 1 of Scale 4 (“feeling safe and in control”) showed the only loading below 0.4 (R²=0.03), suggesting the limited representation of the underlying factor. The item addresses the certainty that only authorized individuals use one’s health care data. According to the local team, health data are typically managed by health care professionals rather than patients, which may have made the item seem irrelevant to participants. Difficulties in interpreting the phrase “who is supposed to use it,” noted during pretests, may also have contributed to the low loading. Overall, the result may reflect the influence of an external construct not captured by the fourth factor. A comparably lower loading (0.46) in the Arabic study may suggest cross-cultural measurement issues of the item [45]. As a sensitivity analysis, we repeated the CFA without item 1 (Multimedia Appendix 3). Model fit was SRMR=0.07, CFI=0.97, TLI=0.97, and RMSEA=0.09. Cronbach α of Scale 4 was 0.59 (95% CI 0.51‐0.67) and McDonald ω 0.70 (95% CI 0.42‐0.81). After extensive discussion, we decided to retain the item despite its limited performance to ensure comparability with other eHLQ versions. The findings suggest that the results from Scale 4 should be interpreted with caution. Furthermore, item 3 on Scale 6 (“access to digital services that work”) showed a comparatively low factor loading (λ=0.44, 95% CI 0.35‐0.52), consistent with findings from the Arabic and Swedish versions (0.42 and 0.35, respectively) [43,45].

In line with the Danish validation study [36], we found strong intercorrelations between eHLQ scales, indicating potential overlap of the scales. In this study, the highest correlations were observed between the means of Scales 3 and 7 as well as between Scales 5 and 7. Kayser et al [36] interpreted high interfactor correlations as reflecting domains that may lie on the same causal pathway while still being conceptually distinct [36]. In the Arabic version, high interfactor correlations led to the combination of Scales 6 and 7 [45]. We interpreted these findings as suggesting a possible general eHealth literacy factor underlying the 7 factors. The better fit of the second-order factor model supports this interpretation and may justify calculating a total eHealth literacy score.

The scales of the Amharic eHLQ demonstrated good internal consistency overall, especially given the low number of items per scale [83]. Both Cronbach α and McDonald ω exceeded 0.70 for all scales except Scale 4 (“feeling safe and in control”), indicating insufficient internal consistency for this scale. This contrasts with previous studies, where Scale 4 showed adequate values (Danish: α=0.86 [36], Swedish: α=0.83 [43], Arabic: ω=0.76 [45]). However, as McDonald ω for Scale 4 was 0.70 and Cronbach α exceeded 0.60, we considered the scale acceptable. Scale 6 (“access to digital services that work”) showed the second lowest values with α=0.72 (95% CI 0.63‐0.77) and ω=0.78 (95% CI 0.72‐0.83). Similarly, in the Danish and Swedish validation study, Scale 6 showed relatively low values (α=0.77 and α=0.82, respectively) [36,43].

Scale 4 of the Amharic eHLQ requires further attention, as it showed weaker internal consistency and included the lowest loading item (item 1). This may indicate cultural mismatches in how the construct of “feeling safe and in control” is represented or reflect limited comprehension because it is rarely addressed in the local context, as noted by the local team. Future studies should further investigate this to rule out the influence of extraneous factors or construct-irrelevant variance. Qualitative approaches may provide deeper insights into how the scale and health data security, in general, are perceived in the Ethiopian context.

The mean total score of the Amharic eHEALS in this study was 20.1 (SD 6.9), which was lower than that reported among patients with chronic conditions at the University of Gondar Comprehensive Specialized Hospital (24.6) [22]. Strong to moderate positive correlations between the eHEALS total score and most eHLQ scales indicate evidence for convergent validity but only limited evidence for discriminant validity. The limited evidence for discriminant validity is likely due to substantial scale intercorrelations. Consistent with our findings, the validation study of the Spanish translation also reported evidence of convergent validity and limited evidence of discriminant validity [41].

Our study has several strengths. The eHLQ is based on a robust framework [35,36] and has demonstrated stable psychometric properties across different cultural contexts [41,44,45]. We followed a rigorous translation process using the Translation Integrity Procedure, which has been successfully applied in other validation studies of the eHLQ [60]. Moreover, this study represents the first validation of the eHLQ in a low-resource country in Africa, providing valuable and novel insights into its applicability and use.

The inclusion criteria of smartphone ownership and internet access possibly limit the generalizability of our study to individuals with lower digital exposure. For this validation study, these criteria were purposefully chosen to ensure that participants had access to digital environments relevant to the construct assessed by the questionnaire. Nevertheless, our sample included a wide range of eHealth literacy levels. Beyond digital access, generalizability is further limited by convenience sampling in 2 health care facilities due to limited access to recruitment sites. Taken together, these limitations may contribute to an unbalanced sociodemographic sample, with an overrepresentation of educated women from urban areas. A context-sensitive application of the tool remains essential. Future studies should employ stratified sampling to ensure adequate representation of men, rural populations, and individuals with lower education levels.

Unlike in the original validation study [36], the Amharic eHLQ was interviewer-administered, which may have influenced participants’ responses. For instance, it could have increased the risk of socially desirable response bias or prevented participants from answering at their own pace [86,87]. To minimize this, we used experienced data collectors and provided them with 1 day of training. Data collectors were informed about the general purpose of the study but not about specific psychometric hypotheses.

Finally, to mitigate a potential unilateral perspective due to the positionality of the first author—based in Germany—and the resulting cultural differences and possible differing understandings of concepts used in the questionnaire, we ensured close collaboration with Ethiopian researchers throughout the entire study period and used a structured translation process [60,88].

Future research should focus on multiple-group CFAs and investigate metric and configural invariance to ensure that comparisons across groups are meaningful.

The Amharic eHLQ can be considered a psychometrically sound instrument for assessing eHealth literacy among patients with internet access in Ethiopia, representing the first validation of the eHLQ in a sub-Saharan African context. Future studies should focus on invariance testing to ensure meaningful group comparisons and provide additional evidence to substantiate the psychometric robustness of Scale 4 (“feeling safe and in control”), as well as qualitative research on local conceptualizations and familiarity with data security. Overall, applying the Amharic eHLQ can provide valuable insights into the experiences of patients with eHealth, inform policymakers and service providers about patients’ needs, and guide the development of targeted eHealth interventions in Ethiopia.