This study is a secondary analysis of data from the 2022 GetCheckedOnline community survey, a cross-sectional survey designed to assess awareness and use of the service in communities where it is available. As part of the survey, we assessed DHL using the eHEALS scale as one of the determinants of awareness and use of the service. GetCheckedOnline is a publicly funded digital sexual health service operated by the British Columbia Centre for Disease Control since 2014 [24,25]. The service is available in nine urban, suburban, and rural communities across British Columbia and provides low-barrier access to sexually transmitted and blood-borne infection (STBBI) testing [24]. It allows users to create an account online, complete a sexual risk assessment, generate a laboratory test requisition, and submit specimens at a designated laboratory, with results accessible online or via public health follow-up [25].

Survey participants were recruited through a combination of online and in-person sampling strategies to ensure diverse representation. Online recruitment involved geo-targeted digital advertisements, local message boards, community-specific Facebook groups and local community organizations’ social media accounts, as well as QR codes placed on outreach materials in community locations. In-person recruitment used venue-based sampling at public locations and events (eg, music festivals, Pride parades, and harm reduction breakfasts). We also conducted snowball sampling facilitated through community leaders and partner agencies. Eligible participants were English-speaking residents of British Columbia aged ≥16 years who reported being sexually active (defined as oral, vaginal, or anal sex with at least one partner in the past year). Individuals who had previously completed the survey or did not reside in BC based on their forward sortation area codes were excluded.

The survey was conducted between July and September 2022 and was collaboratively designed by the GetCheckedOnline team in consultation with a Community Advisory Board, which included individuals with lived experience of STBBI testing within BC and those working with community organizations serving populations most affected by STBBIs. The questionnaire collected information on DHL, sociodemographic characteristics, digital technology access and use, and experiences with sexual health and STBBI testing. DHL was assessed using the eHEALS scale. Responses were collected on a 5-point Likert scale, with higher scores indicating greater DHL.

Descriptive analyses were conducted to summarize participants’ characteristics, including age, gender, ethnicity, education, income, and digital access. The distributions of eHEALS item responses were assessed using medians and IQRs, given the ordinal nature of the data. Item-level correlations were examined using polychoric correlations, which better capture relationships between ordinal variables than Pearson correlations. A correlation matrix was visualized using a heat map to explore patterns of association between eHEALS items. EFA was conducted to determine the underlying structure of the eHEALS scale.

Sampling adequacy was assessed using the Kaiser-Meyer-Olkin (KMO) measure, with values above 0.80 indicating strong factorability. Bartlett’s test of sphericity was performed to assess whether item correlations were sufficiently large for factor analysis. Factor extraction was conducted using polychoric correlation matrices and the weighted least squares means, and variance adjusted estimator, which is appropriate for ordinal data [26]. The number of factors was determined using parallel analysis, eigenvalues, and the scree plot, and factor loadings were examined using oblique rotation to account for potential correlations between factors [27]. CFA was conducted using structural equation modeling to validate the factor structure identified in the EFA.

Multiple models were tested, including a one-factor model and three-factor models based on theoretical and empirical considerations. Model fit was evaluated using several indices, including the comparative fit index (CFI) and Tucker-Lewis index (TLI), with values above 0.95 indicating good fit [28]. The root-mean-square error of approximation (RMSEA) and its 90% CI were used to assess model parsimony, with values below 0.08 considered acceptable. The standardized root-mean-square residual (SRMR) was also examined, with values below 0.05 suggesting good fit [28]. Where there were significant cross-loadings, we examined model fit indices and selected models with better fit indices and optimal factor loadings. Choices were informed by theoretical plausibility and evidence from previous studies [11,14]. Missingness across eHEALS items ranged from 12.61% to 13.46%. Patterns and mechanisms of missingness were examined prior to analysis. The Little MCAR test was conducted to evaluate whether data were missing completely at random. The test indicated significant deviation from the MCAR assumption (χ²₃₅₂₇= 5702, P<.001), suggesting that missingness was not completely random. As missingness was therefore unlikely to be completely random, multiple imputation using chained equations was applied under a missing-at-random assumption to minimize potential bias and preserve statistical power [29-31]. Sensitivity analyses were performed to compare results from the imputed dataset with a complete-case analysis.

Reliability and validity analyses were conducted to assess internal consistency and construct validity. Polychoric ordinal α and McDonald ⍵ were calculated for each factor to assess internal consistency, with values above 0.70 indicating acceptable reliability. Construct validity was evaluated through convergent validity, assessed using average variance extracted (AVE), with values above 0.50 suggesting good convergence [32]. Discriminant validity was assessed by ensuring that AVE values exceeded the squared correlations between factors. All analyses were conducted using R (version 4.3.1), with Lavaan used for structural equation modeling, psych for EFA and reliability analyses, and Corrplot for correlation visualizations [33,34].

This study was reviewed and approved by the University of British Columbia Behavioral Research Ethics Board (REB#H25-01028) allowing for secondary analysis of previously collected anonymous data without additional consent. The original 2022 community survey data obtained informed consent from all participants. Survey participants were offered the option to enter a draw for one of five CAD $100 (US $71) Visa gift cards by providing separate contact information that was not linked to their survey responses. The dataset used for this analysis was fully deidentified before access, and no personal identifiers were available to the research team. No identifiable images or other personal data are included in this manuscript or its supplementary materials. The study adhered to STROBE (Strengthening the Reporting of Observational Studies in Epidemiology) checklist for cross-sectional studies (Multimedia Appendix 1), ensuring transparency and rigor in reporting [35].

Among 2720 initial survey respondents, 1657 met the inclusion criteria, with a mean age of 33.0 (SD 11.77) years. Among these, 47.3% (n=784) identified as women, 14.5% (n=240) as gender identities characterized as gender diverse, 30.4% (n=503) as belonging to a racialized minority group, 49.5% (n=820) as a sexual minority, and 39.4% (n=653) as straight. Regarding education, 29.9% (n=495) had a postsecondary education and 32.2% (n=534) had a bachelor’s degree or higher. Among respondents 23.5% (n=389) earned <CAD $20,000 per annum while 80.5% (n=1334) reported easy access to the internet (Table 1).

The mean scores for the eight eHEALS items ranged from 3.62 (SD 0.99) to 4.01 (SD 0.83) and a median score of 4.0 out of a possible score of 5, indicating most respondents had moderate to high agreement with digital health literacy statements (Table 2 and Figure 1). The internal consistency of the scale was excellent, with a polychoric ordinal α of 0.92 and an average inter-item correlation of 0.60, supporting the reliability of the measure for ordinal data (Figure 2).

The KMO test demonstrated an overall sampling adequacy for factor analysis of 0.89 with values for individual items ranging from 0.86 to 0.91. The Bartlett test of sphericity revealed significant findings (χ²₂₈=7222.8, P<.001), supporting the suitability of factor analysis. EFA supported a three-factor solution with eigenvalue of >1 as demonstrated by a parallel analysis scree plot (Figure 3), explaining 65.7% of the variance within the dataset. We also noted potential cross-loading with item 8 loading across 2 out of the 3 factors (Table 3).

In CFA, the 1-factor model (originally proposed by Norman and Skinner [4]) exhibited a poor fit to the data (χ²₂₀=849.208, P<.001, RMSEA=0.158, SRMR=0.072, CFI=0.987, TLI=0.981; Table 4). The three-factor model proposed by Sudbury-Riley et al [11] revealed improved fit to the data (χ²₁₇=420.215, P<.001, RMSEA=0.127, SRMR=0.041, CFI=0.935, TLI=0.894). However, the EFA-informed structure (model 3), assuming continuous variables, resulted in a substantial improvement in fit indices (χ²₁₆=163.558, P<.001, RMSEA=0.077, SRMR=0.025, CFI=0.977, TLI=0.961). The final model (model 4), incorporating the same EFA-informed three-factor structure but accounting for the ordinal nature of eHEALS responses, exhibited the best model fit (χ²₁₇=71.680, P<.001, RMSEA=0.059, SRMR=0.026, CFI=0.969, TLI=0.948), supporting the appropriateness of a three-factor solution while treating eHEALS items as ordinal data.

The final three-factor model of the eHEALS scale consisted of the following domains, which we describe as Information Navigation, Resource Appraisal, and Confidence in Use. The standardized factor loadings from CFA are summarized in Table 5. Information Navigation reflects participants’ ability to locate and access online health resources effectively with factor loadings for this domain ranging from 0.765 to 0.917 (ie, strong relationships between items and the underlying construct). Resource Appraisal assesses respondents’ capacity to evaluate the reliability and relevance of online health information. Items within this factor demonstrated factor loadings ranging from 0.825 to 0.892, reflecting good construct validity. Confidence in use represents the ability to apply digital health information confidently in decision-making, with factor loadings of 0.803 for both corresponding items. The standardized loadings suggested that all items load significantly onto their respective latent factors, demonstrating the validity of the three-factor structure. Additionally, all factor loadings exceeded 0.70, which is considered strong evidence for convergent validity.

Table 6 summarizes the internal consistency of the three factors using composite reliability and AVE. The composite reliability values ranged from 0.784 to 0.900, exceeding the recommended threshold of 0.70, suggesting adequate reliability. The AVE values assessing the proportion of variance captured by each latent factor relative to measurement error, ranged from 0.503 to 0.738, supporting good convergent validity.

In this study, we identified and validated a multidimensional structure of the eHEALS, confirming that based on the existing eHEALS questionnaire, DHL is best conceptualized through three distinct but related factors: Information Navigation, Resource Appraisal, and Confidence in Use. Information Navigation describes an individual’s ability to locate and retrieve health-related information online, Resource Appraisal reflects their capacity to critically assess the quality and credibility of digital health content, and Confidence in Use captures their self-assurance in using online information for health decision-making. The final three-factor model demonstrated excellent fit, construct validity, and reliability reinforcing the utility of this revised eHEALS in assessing DHL. The strong fit indices for the three-factor model—particularly when accounting for the ordinal nature of the eHEALS questionnaire—highlight the importance of appropriately modeling the categorical nature of DHL survey items. The validation of this refined factor structure offers a more precise measurement framework for assessing DHL across diverse populations.

Previous research has largely treated eHEALS as a unidimensional scale, measuring digital health literacy as a singular construct [4,12]. However, emerging evidence suggests that DHL is multifaceted, requiring distinct skills for accessing, understanding, evaluating, and applying health information in online contexts [5,11,16,36,37]. Our results align with and expand upon prior studies that have questioned the original unidimensional structure of eHEALS, supporting the need for a revised framework that captures different facets of DHL [11,36].

Sudbury-Riley et al [11] proposed an alternate three-factor structure for eHEALS based on socio-cognitive and self-efficacy theory to suggest DHL consists of awareness, skills, and evaluation capabilities [11]. Others have suggested two- or three-factor structures for eHEALS [16,17,38]. Our model advances multidimensional conceptualizations of the eHEALS by explicitly accounting for the ordinal nature of the scale, particularly in the context of widespread digital transformation, which has contributed to nonnormal response distributions. The latent factors identified in this analysis reflect key components of Bandura’s Social Cognitive Theory, which emphasizes the role of personal capabilities, environmental factors, and social influences in shaping knowledge acquisition [5,39]. Our results also align with a recent conceptual model of DHL, which identified four main attributes including goal-driven regulation, information processing, usage, and communication [37]. Our findings indicate that information navigation, resource appraisal, and confidence in use are distinct yet interrelated constructs, supporting the view that DHL is not simply about information retrieval but extending to include the ability to critically engage with and apply online health information [5,37]. Interventions to promote DHL need to not only focus on the peoples’ self-efficacy and personal capabilities through tutorials and demonstrations, but also community-level and social factors that can promote DHL. For example, at the interpersonal level, there is emergent evidence of the impact of peer support networks in enhancing marginalized communities’ collective trust in digital health technology which ultimately influences DHL [40,41]. At the community-level, public goods like library services have been leveraged to promote training and vocational skills that can contribute to DHL [42]. Beyond these efforts must address upstream factors including digital access and online regulatory policies that influence community trust [40,41].

This study validates the multidimensional factor structure of eHEALS using advanced psychometric techniques and multiple imputation within a general population as compared to others that have not recognized the ordinal nature of the survey and have been conducted in subsets of the population (eg, among older adults) limiting their generalizability [11,17]. By leveraging EFA ordinal CFA and polychoric correlations, we provide strong empirical evidence for a revised three-factor structure of the eHEALS scale that accounts for the ordinal structure of the data. This revised structure provides greater conceptual clarity and measurement precision, making it particularly useful for researchers, clinicians, and public health practitioners seeking to assess DHL.

Within the context of societal digital transformation, there has been increasing recognition of the need for improved measures of DHL to guide public health advocacy, policy, and action at multiple levels [2,8,43,44]. DHL is now considered a fundamental determinant of health, shaping individuals’ ability to leverage health information in an increasingly digital world [8,40,41]. At a population level, a more nuanced understanding of DHL can inform better-targeted interventions by identifying specific gaps in information-seeking skills, critical evaluation of online content, or confidence in applying health information as described above. Traditionally, digital literacy interventions have focused on improving access and navigation; yet, emerging evidence suggests that developing appraisal skills and trust in credible sources is equally critical. These skills have become particularly important in light of the proliferation of online mis- and disinformation, which has significantly increased since the COVID-19 pandemic, fueling vaccine hesitancy and the re-emergence of vaccine-preventable diseases such as measles [9,45].

At a program level, this validated three-factor structure of DHL can serve as a planning and evaluation tool for digital health programs, enabling more precise assessment of intervention effectiveness and guiding ongoing program adaptations [46]. This is particularly relevant for public health initiatives targeting equity-deserving populations, who may experience unique barriers to engaging with digital health tools [46,47]. For instance, interventions aimed at populations with poorer resource appraisal skills may need to incorporate structured training on misinformation detection, fact-checking strategies, and trust-building in digital health sources [10]. To maximize their impact, adapted DHL measures must be considered in national surveys like the Canadian Community Health Survey [48].

Future research should examine the generalizability of this three-factor model across diverse populations, including different age groups, cultural and linguistic contexts, and health conditions. Longitudinal studies are particularly needed to assess how DHL evolves over time, its impact on real-world health behaviors, and its role in mediating or moderating sociodemographic disparities in access to and use of digital health interventions. Furthermore, as with increasingly digital-first health services and with increasing integration of artificial intelligence and algorithm-driven health information, refining DHL measures will be essential [10,49,50]. This revised scale serves as a foundation for developing more precise and contemporary assessments that capture the complexities of DHL in an era of rapidly advancing health technologies.

This study has several notable strengths. First, we conducted a rigorous psychometric evaluation, incorporating both exploratory and confirmatory factor analysis, to validate the revised eHEALS structure. Our approach leveraged polychoric ordinal factor analysis, which appropriately models the categorical nature of eHEALS items, ensuring more accurate measurement. The large and diverse sample enhances the generalizability of our findings across different populations. Recruiting over 60% of participants via in-person channels contributed to ensuring a diverse sample.

However, some limitations should be acknowledged. Our study relied on self-reported measures of DHL, which may be subject to response bias. This is a widely debated issue, given evidence suggesting that survey respondents may overestimate their DHL [51,52]. The original survey was limited to English-speaking participants and relied on a convenience sample of individuals who were willing to participate, introducing potential self-selection bias and limiting the generalizability of the findings. Therefore, although the three-factor model demonstrated excellent model fit, further validation across diverse populations, including non–English-speaking groups and different health care contexts, is needed to ensure generalizability. This is especially relevant as our survey targeted populations accessing testing for sexually transmitted infections with estimated DHLs being significantly higher than the Canadian average [22]. While most fit indices were excellent, slightly lower construct reliability for the factor Confidence in Use suggests the need for more reliable scales that are dependent on more refined items. Moreover, given the rapid evolution of digital technologies, findings reinforce the need for continued refinement of DHL measures to better capture emerging competencies required for navigating modern digital health ecosystems.

This study validates a revised three-factor structure of the eHEALS, confirming that DHL comprises three distinct but interrelated competencies: Information Navigation, Resource Appraisal, and Confidence in Use. These findings enhance conceptual clarity in measuring DHL, providing stronger links to the DHL literature and an empirical foundation for future research and intervention development. As digital health platforms continue to play an increasingly central role in health care and public health, the ability to accurately assess and target DHL is essential for promoting informed health decision-making. The validated three-factor model offers a more precise tool for identifying specific gaps in DHL, enabling tailored interventions that equip individuals with the skills needed to navigate, evaluate, and apply digital health information effectively. However, further research is required to explore how these factors influence access to digital health services, while ongoing efforts must focus on developing updated scales that account for emerging technological advancements, including the growing influence of artificial intelligence and social media in health information ecosystems.