The DHL Questionnaire for Stroke Survivors was developed using a sequential, exploratory, mixed methods approach (Figure 1), and the study period ran from September 15, 2023, to April 28, 2024. The design involves an initial qualitative phase aimed at generating the item pool and a subsequent second phase aimed at testing items using quantitative techniques (psychometric evaluation).

We used convenience sampling, and all participants gave informed consent at the beginning. Following the survey, professional researchers conducted in-person examinations with participants. Content validation was conducted among stroke survivors recruited from the Department of Neurology and Department of Neurosurgery at the Fifth Affiliated Hospital of Zunyi Medical University, as well as among experts in the field of neurology. The inclusion criteria were defined as being aged 18 years or older, owning a smartphone, and having the ability to read and understand Chinese. Individuals with mental disorders were excluded from the study.

The field test population included survivors with self-reported strokes. Participants were aged 18 years or older and could self-complete the DHL Questionnaire for Stroke Survivors in Chinese. A minimum sample size of 250 respondents was prespecified for field testing. In Rasch analysis, a sample of 250 respondents provides 99% confidence that item calibrations and person measures are stable within a SD of 0.50 logits [19]. Psychometric procedures were conducted to determine the final items and cutoff scores of the DHL Questionnaire for Stroke Survivors.

In this research, the determination of sample size was guided by principles for conducting exploratory factor analysis (EFA). Academic literature [20] offers various heuristics for sample size estimation in EFA, with absolute minimums ranging from 100 to 250 participants [21]. Some sources suggest larger samples of 300, 500, or more [22,23]. A ratio-based approach is often recommended, where the number of participants (N) to the number of variables (p) ideally should be 5:1, meaning 5 participants for each variable [24].

For both EFA and confirmatory factor analysis (CFA), 2 separate, independent samples are necessary. The minimum required sample size for this study was calculated to be 228 participants, considering a 20% allowance for missing data. It is also noted that a sample size ranging from 30 to 200 is considered sufficient for conducting a Rasch analysis [25].

The lily model, grounded in social cognitive and self-efficacy theories, prioritizes enhancing capabilities and confidence over direct skill measurement to foster behavioral change and skill development [9]. We adopted the lily model to identify 3 key domains of eHealth literacy: information acquisition, evaluation, and application abilities. At its heart are 6 core skills (or literacies): traditional literacy, health literacy, information literacy, scientific literacy, media literacy, and computer literacy. Using the metaphor of a lily, the petals (literacies) feed the pistil (eHealth literacy), and yet the pistil overlaps the petals, tying them together. The lily model of eHealth literacy is used, because it is comprised of multiple literacy types, including an outline of a set of fundamental skills patients require to derive direct benefits from eHealth. A profile of each literacy type with examples of the problems patient-clients might present is provided along with a resource list to aid health practitioners in supporting literacy improvement with their patient-clients across each domain.

For item generation during the development of the new scale, it was important to pool all attributes reflecting the construct being measured. A literature review was used as the source in this study. For the comprehensive literature review, a matrix table was constructed based on the abovementioned DHL conceptualization. The first column on the left of the matrix was dedicated to the posited abilities and skills (acquire, evaluate, and apply). Beginning with the second column, the matrix incorporated specific items. From the third column onward, relevant references were systematically listed.

Content validation was conducted through a web-based expert survey designed to explore the relevance and clarity of the items and to identify items relevant to clinical concerns about DHL. The Item-Level Content Validity Index value for each item was calculated based on the proportion of expert ratings of the item’s relevance (modified κ value). Specifically, a modified κ value in the range of 0.4-0.59 indicates fair content validity, 0.60-0.74 indicates good, and ≥0.74 indicates excellent [26]. We relied on the content validity index values of the items to identify those candidates that required further in-depth exploration of their relevance and comprehensibility with stroke survivors.

Cognitive debriefing interviews with stroke survivors were conducted face-to-face with the aim of assessing the relevance, comprehensiveness, comprehensibility, and acceptability of the DHL Questionnaire for Stroke Survivors. Verbatim transcripts of the interviews were qualitatively analyzed using thematic analysis. These stroke survivors were also invited to point out other DHL components that may have been omitted from the initial item pool.

A convenience sample of 418 participants was recruited from the Departments of Neurology and Neurosurgery of the Fifth Affiliated Hospital of Zunyi Medical University in China from December 1, 2023, to April 28, 2024.

Of the stroke survivors who completed the study survey, 20 were asked to complete the DHL Questionnaire for Stroke Survivors again 2 weeks later.

Content validity indices were calculated for each item based on expert ratings of relevance. The Item-Level Content Validity Index scores for these items ranged from 0.40 to 1, with 20 (80%) of the draft items rated as good or excellent.

The experts’ review (n=20) of the item pool informed changes to the wording of items to improve clarity (Multimedia Appendix 2).

Cognitive debriefing was established through cognitive debriefing interviews with 15 Chinese stroke survivors aged 18 years or older. The participants’ ages ranged from 53 to 72 years, with 7 (46%) participants being male and 8 (53%) participants being older than 65 years. The average monthly income for the participants was between CNY ¥3000 and ¥5000. A total of 7 (47%) participants reported living with a spouse, while 11 (73%) participants had been living with their condition for over a year. In terms of daily activities, 13 (87%) participants demonstrated a mild level of dependence, while 2 (13%) participants were moderately dependent. It is worth noting that 3 (20%) participants were experiencing the illness for the first time.

Thematic analysis was used to classify problems with the relevance, comprehensiveness, clarity, and acceptability of the draft items. The key themes and exemplar quotations from the thematic analysis can be found in Multimedia Appendix 3. The draft DHL Questionnaire for Stroke Survivors consisted of 20 items, organized into 3 domains based on a priori theoretical classification. Each domain was constructed as an independent scale. We used a 5-point Likert scale for assessment (1=strongly disagree, 2=disagree, 3=undecided, 4=agree, and 5=strongly agree). Higher scores indicated a higher level of DHL.

A total of 480 questionnaires were distributed, yielding 450 completed responses, resulting in a response rate of 93.75%. Of these, 32 submissions were deemed incomplete and were therefore excluded from the analyses. Consequently, the final sample comprised 418 complete responses, achieving a completion rate of 100%. The ages of the respondents varied from 33 to 90 years, with a mean age of 65.25 (SD 12.02) years. The sample consisted of 167 female respondents, accounting for 40% of the total. Table 1 provides further details.

During the scale refining process, we first used the critical ratio method, sorted scale scores to identify high and low groups, and calculated critical ratio values using t tests. Item 19 was eliminated because its critical ratio value fell below the threshold of 3. Furthermore, Spearman rank correlation analysis was performed, resulting in the elimination of item 18 due to a correlation coefficient of less than 0.4. Following these analyses, the scale was simplified to include 17 items.

Following an EFA on the 17-item subset, the KMO score was 0.901, which was above the acceptable threshold of 0.7, suggesting that the data were appropriate for component analysis. The Bartlett test of sphericity showed significant results (χ²₁₃₆=1372.687, P<.001). PCA showed 3 variables with eigenvalues greater than 1, accounting for 69.212% of the variance, showing excellent explanatory power (Table 2). The scree plot confirmed the retention of 3 factors, as the line leveled off after the fourth factor (Multimedia Appendix 4). The scale’s construct validity was confirmed by factor loadings over 0.5 and no cross-loadings above 0.5. The items were divided into 3 categories: “acquisition ability,” “evaluation ability,” and “application ability.”

CFA indicated an excellent model fit, with indices (χ²₄=1.669; root mean square error of approximation=0.047; Tucker-Lewis Index=0.973; Comparative Fit Index=0.977) aligning with conventional criteria for a good fit. Composite reliability estimates varied from 0.861 to 0.932, and average variance extracted scores were between 0.555 and 0.664, indicating convergent validity. Furthermore, the factor loading values, which are shown in Multimedia Appendix 5, varied from 0.616 to 0.937, supporting convergent validity.

The value for Cronbach α, as further evidence of internal consistency reliability, was 0.937. The coefficient of test-retest reliability was 0.941, indicating a high degree of consistency in the measurements over time.

The correlation coefficient between the DHL Questionnaire for Stroke Survivors and eHEALS was 0.853 (P<.001).

To evaluate the functionality of the rating scale, we reviewed the Winsteps 3.66.0 output tables and item category probability curves for the 17 items.

After optimization of the response scales, we conducted a Rasch analysis to report the finalized characteristics of the DHL questionnaire for stroke survivors. The DHL Questionnaire for Stroke Survivors consists of 15 items. Multimedia Appendix 6 lists the items included. The Chinese version of the DHL Questionnaire for Stroke Survivors is provided in Multimedia Appendix 7. The scale meets the unidimensional criteria of the Rasch model. The first residual logarithm of the PCA of residuals ranged between 1.4 and 1.8. Item mean squared error values ranged from 0.71 to 1.38 logits, and equipment mean squared error values ranged from 0.67 to 1.75 logits. Items 4 and 7 showed ambiguity in the model. Ultimately, 15 items met the acceptable item fit criteria for the Rasch model.

Four items showed local item dependency, with residual correlation values >0.4 (range 0.44-0.59). In practical terms, a degree of local dependency is always observed in empirical data; therefore, it is necessary to consider the implications for content validity before proceeding with item removal [34]. After a qualitative appraisal of the 4 dependent pairs, we retained all items to ensure a comprehensive assessment of DHL in stroke survivors.

Last, we evaluated item separation, person reliability and separation, and internal consistency reliability. The scale had mean person ability values within a SD of 1.0 logits of mean item difficulty. Person ability ranged from −3.2 to 2.8 logits. Multimedia Appendix 8 shows the item person maps for the DHL Questionnaire for Stroke Survivors. The item separation index was 3.81 (reliability 0.94) and the individual separation index was 2.91 (reliability 0.89).