A cross-sectional study was conducted in September 2023 to validate the scale. A randomized cluster sampling approach was used, with an average of 60 adult inpatients (excluding pediatrics) recruited from each of 9 wards in a Grade A tertiary hospital in Hunan, China, resulting in a total of 532 participants. The scales were distributed in hard copies by the first author (XYF) and two trained researchers (JJ, MYL). Confirmatory factor analysis (CFA) was used to test the alignment of the scale structure with theoretical assumptions. Model fit was evaluated using a range of indices, such as the goodness-of-fit index (GFI), with values closer to 1 indicating better construct validity [28]. Convergent validity was assessed using average variance extracted (AVE) and composite reliability (CR), with AVE >0.50 and CR >0.70 considered acceptable. Discriminant validity was confirmed if the square root of AVE exceeded interdimension correlation coefficients [29].

Content validity was assessed using the content validity index (CVI) derived from Delphi ratings for relevance. Item-level CVI (I-CVI) values ≥0.78 and scale-level CVI (S-CVI) values ≥0.90 were considered acceptable [30]. The I-CVI was calculated by dividing the number of experts who scored 3 or 4 by the total number of experts, while the S-CVI was determined by averaging the I-CVIs of all the items [30]. Criterion validity was evaluated by correlating the AIPeHLS with the Chinese version of eHEALS translated by Guo et al, which was validated among 110 high school students with a Cronbach α of .913 [31]. A higher correlation coefficient indicated stronger criterion validity.

Cronbach α and McDonald omega (ω) coefficients were calculated for the entire scale and its dimensions, with values ≥0.80 indicating good reliability [26]. Finally, the scale was divided into 2 halves, and the correlation between subscale scores was computed for evaluating split-half reliability, with coefficients ≥0.80 considered good reliability [26]. A detailed flow diagram for the development and validation process is provided in Multimedia Appendix 1.

Ethical approval was obtained from the Research Ethics Board of School of Nursing, Central South University, Hunan, China (No. E202373). Informed consent was obtained from all participants enrolled in this study. All patient data were anonymized, with unique IDs assigned to each participant. The data were securely stored in a password-protected database, accessible only to authorized personnel. Patients were informed about data collection and usage and could withdraw at any time without consequences. As a token of appreciation, each participant received a small gift.

A comprehensive review of 934 articles related to eHL was conducted, resulting in the inclusion of 19 studies for item pool development, comprising 8 Chinese and 11 English articles (Multimedia Appendix 1). An initial pool of 53 items was generated based on this review.

To refine the item pool, a 2-round Delphi consultation was carried out with 18 experts from 12 tertiary-level general hospitals, 2 higher education institutions, and 1 national academic organization across 12 provincial-level regions, including Henan, Jilin, Gansu, Shandong, Sichuan, Hunan, Guangdong, Shaanxi, Liaoning, Shanghai, Beijing, and Xinjiang Uygur Autonomous Region. The panel consisted of 14 health care professionals and 4 experts in computer science and engineering. All participants held at least a bachelor’s degree, with approximately 90% holding master’s degrees or higher. Furthermore, nearly 90% of the experts were at vice senior or senior professional levels, and 95% had more than 10 years of working experience, ensuring high expertise and credibility in their feedback (Table 1).

The response rates for both rounds of consultation were 100%, demonstrating positive engagement. Expert authority, quantified by the authority coefficient, was 0.864, indicating high reliability of the consultation results. Item importance scores averaged >3.50 across both rounds, with full-score rates exceeding 20% for 90.57% and 100% of items in the first and second rounds, respectively. CVs for 93.22% and 96.08% of items were <0.25 across the 2 rounds, reflecting consensus among experts. The Kendall W coefficients for both rounds were statistically significant (P<.001) with values of 0.249 (χ²=262.340) and 0.146 (χ²=131.317), respectively. Based on expert feedback and statistical evaluation from the first round, we modified 22 items, added 3 new items, removed 11 items, and 8 items were merged appropriately. In the second round, 16 items were modified and 1 item was removed, leading to a finalized pool of 44 items distributed across 6 dimensions for the initial version of APIeHLS (Multimedia Appendix 1).

In the critical value analysis, independent sample t tests revealed statistically significant differences (P<.001) between the high- and low- scoring groups for all items and confirmed their discriminatory power (Table 2). The item-total correlation coefficients for each dimension ranged from 0.843 to 0.943 (traditional literacy), 0.745 to 0.905 (information literacy), 0.895 to 0.936 (media literacy), 0.929 to 0.971 (health literacy), 0.832 to 0.881 (computer literacy), and 0.909 to 0.947 (scientific literacy) (Table 2). The correlation coefficients between each dimension and the total score ranged from 0.685 to 0.848, all exceeding the threshold of 0.400 and demonstrating statistical significance (P<.001). The initial scale demonstrated excellent internal consistency, with a total Cronbach α of .959 and subscale values ranging from 0.952 to 0.975. CITCs for all items exceeded 0.400, and the deletion of any item did not significantly improve the Cronbach α, confirming that all items contributed to the overall measurement consistency (Table 2).

The KMO value for the initial scale was .921, and the Bartlett test of sphericity yielded a χ² value of 5871.995 (P<.001), confirming the suitability of the data for factor analysis [26]. Six factors were extracted using the maximum variance method, with eigenvalues of 9.786, 6.668, 6.264, 5.820, 4.252, and 3.562, respectively, accounting for a cumulative variance contribution of 82.616% (Table 3). All factor loadings were ≥0.400, and no cross-loading was observed, supporting the clarity and distinctiveness of the dimensions (Table 4). The scree plot is illustrated in Figure 1. The finalized AIPeHLS included 44 items rated on a 5-point Likert scale from 1 (strongly disagree) to 5 (strongly agree) across 6 dimensions.

CFA was conducted to validate the factor structure identified in the EFA. Standardized path models were constructed based on data from 532 participants using AMOS 26.0. Fit indices demonstrated acceptable model fit: χ²=1974.654 (df=887) , GFI=0.854, AGFI=0.837, root mean square error of approximation=0.048, RMR=0.052, comparative fit index=0.957, normed fit index=0.925, and IFI=0.957, which met the criteria of <3.000, >0.900, >0.900, <0.050, <0.050, >0.900, >0.900, and >0.900, respectively. These results confirmed the structural validity of the scale (Figure 2). Convergent validity was supported by standardized factor loadings >0.700, AVE values ranging from 0.695 to 0.835 (>0.500), and CR values between 0.948 and 0.971 (>0.700) (Table 5). Discriminant validity was demonstrated by AVE square roots exceeding interdimensional correlation coefficients, which ranged from 0.354 to 0.466, indicating that each dimension was distinct and unidimensional (Table 6).

Content validity, assessed using expert-rated relevance scores, showed excellent results with an S-CVI of 0.961 and I-CVIs ranging from 0.889 to 1.000 (Table 7). Criterion-related validity was evaluated using Spearman rank correlation coefficients, resulting in a total coefficient of 0.992 (All P<.001), and subscale correlations ranged from 0.607 to 0.785. This revealed a strong correlation between the AIPeHLS and the external criterion, supporting the scale’s applicability and relevance (Table 8).

The scale demonstrated strong reliability, with a total Cronbach α coefficient of 0.965. The subscales can be used separately, with Cronbach α coefficients of 0.961 (traditional literacy), 0.971 (information literacy), 0.956 (media literacy), 0.964 (health literacy), 0.948 (computer literacy), and 0.953 (scientific literacy). Also, the omega coefficient for the total scale was 0.962, with subscale values ranging from 0.948 to 0.971, indicating high internal consistency. Split-half reliability, assessed using the Spearman–Brown correlation method, yielded a total reliability coefficient of 0.791, with subscale values consistently above 0.960. The final version of the AIPeHLS is available in Multimedia Appendix 2.

This study successfully developed and validated the AIPeHLS, a comprehensive and psychometrically robust instrument tailored to measure eHL in adult inpatients. Grounded in the Lily model, the AIPeHL comprises 44 items that reflect the diverse and evolving competencies required to navigate the digital health landscape, spanning from Web 1.0 to Web 3.0. These items take advantages over existing eHL tools [17-19] by incorporating advanced competencies such as data security, technical problem-solving, and personalization, which are increasingly essential in health care systems that become more digitized and patient-centric. The validation process of AIPeHLS adhered to rigorous psychometric standards to ensure its reliability, validity, and applicability across clinical and research settings. Importantly, the AIPeHLS represents an innovative and forward-looking solution tailored to hospitalized adult inpatients, addressing a critical gap left by previous tools that were either too broad for the general public or targeted at digitally adept younger audiences. This study provides insights for health care providers to better understand how patients make health-related decisions based on their eHealth competencies and develop targeted interventions, while researchers can use the scale to explore opportunities for optimizing user-centered health care solutions.

The development of the AIPeHLS was guided by the Lily model, which conceptualizes eHL as comprising 6 interrelated dimensions. This framework informed the item generation, ensuring that the scale captures both foundational and advanced eHL skills relevant to the digital health challenges faced by inpatients. In this study, a comprehensive literature review and expert consultations were conducted to enhance the relevance and inclusivity of the items.

Within the Lily model, traditional (A), information (B), and media literacy (C) are categorized as analytic components that are foundational and applicable across contexts [1]. For traditional literacy, in addition to a general emphasis on reading and writing (A1, A3) [22], we included more real-world scenarios, such as using voice interactions in instant messaging apps (A2) [21], articulating health concerns (A5) [22], and sharing experiences with peers and caregivers in person or via online health communities (OHCs) (A6) [21]. These items are necessary to understand patients’ engagement in digital health environments and to identify potential barriers to interaction. Numeracy, a critical component of traditional literacy but often overlooked in previous eHL assessments, was also given particular emphasis in this study (A4) as it is essential for inpatients to interpret medication dosages, understand cost-related information, and manage complex treatment regimens. Information literacy refers to the ability to search for (B7, B8), filter (B16), and evaluate (B10, B11, B12, B13) health-related information proficiently [17-22]. The AIPeHLS expands this definition to account for emerging concerns unique to the digital age, such as identifying commercial biases in online content (B15) [21,22], assessing data privacy risks (B17, B19), and managing personal account security (B18). These competencies are increasingly relevant as AI technologies gain prominence in health care, with the percentage of AI-generated information projected to increase from 1% of all human data in 2022 to 10% by 2025 [32]. Despite breakthroughs in the medical field, unsupervised AI tools can potentially generate hallucinations that impact patient decision-making and even lead to unintended negative consequences, such as biased treatment recommendations and inappropriate mental health advice [33]. To mitigate these risks, the scale also evaluates patients’ ability to find or verify online information through consultations with health care professionals (B9, B14) [18], who remain the most trusted sources for validating the credibility and accuracy of health information. This dual focus on independent information evaluation and professional consultation reflects the evolving interaction patterns between patients and health care providers in the digital era. Media literacy is particularly critical in the social media context, where misinformation can spread rapidly, as seen during the pandemic. In this dimension, the AIPeHLS assesses patients’ ability to critically appraise, question, and correct misinformation encountered online (C20, C21, C22). We also considered behaviors related to ethical content sharing, such as posting illness diaries (C23) [20], avoiding spreading unverified content (C24, C25) [21] and preventing plagiarism (C26) [21], providing insights into patients’ roles as both consumers and disseminators of digital health information.

In contrast, health (D), computer (E), and scientific literacy (F) are categorized as context-specific components that rely on more situation-specific skills [1]. Specifically, health literacy empowers patients to use eHealth tools to address health-related issues promptly. Therefore, knowledge of medical terminology (D27) [19], treatment options (D29), and awareness of changes in health conditions (D28) were considered in this dimension. It also evaluates patients’ ability to leverage eHealth tools for decision-making (D31, D32) [22] and self-management (D30) [21], aligning with the goals of eHealth interventions. Computer literacy involves the technical skills required to operate digital devices and navigate innovative tools. Research showed that a lack of experience in using technologies could limit patients’ ability to engage with and benefit from digital health services [34]. The AIPeHLS addresses this gap by assessing familiarity with technological terms (E33), basic operations (E34) [22], safety measures (E35), and problem solving (E36). Moreover, this scale assesses the ability to select (E38), use (E37), and adapt to eHealth tools (E40) [19] for proactive health promotion, such as tracking medical reports, monitoring lifestyle factors (eg, sleep, exercise, and nutrition), and formulating health plans to achieve specific goals [21]. Finally, scientific literacy plays a crucial role in enabling patients to comprehend the scientific foundations underlying health recommendations and critically evaluate the credibility of eHealth tools. For individuals without a background in scientific education, interpreting research-based health information presented online can be particularly challenging [1]. In this study, we assessed whether patients understand that suggestions from eHealth tools are based on time-sensitive research findings (F41, F42) and whether they can recognize the functionalities and limitations of these tools (F43, F44). This focus is novel and significant for selecting eHealth tools rationally and objectively, empowering patients to navigate the complexities of digital health with confidence and competence.

The construct validity of the AIPeHLS was determined through both EFA and CFA. The EFA results demonstrated a clear factor structure, with each item loading strongly onto its respective dimension and minimal cross-loadings. Subsequent CFA confirmed the factorial composition of AIPeHLS, with commonly used fit indices, such as χ², root mean square error of approximation, comparative fit index, normed fit index, and IFI, meeting or exceeding recommended thresholds. Convergent and discriminant validity were confirmed by the AVE, with values above 0.500 and its square root values greater than interdimensional correlation coefficients. The content validity of the APIeHLS was ensured through a rigorous item development process, with expert panels evaluating the importance, relevance, clarity, and representativeness of each item, achieving CVI values exceeding 0.800 across all dimensions. Criterion-related validity was supported by its high correlation with the Chinese version of eHEALS, a widely accepted instrument, although it has some limitations in capturing the most authentic eHL performance among patients. Regarding reliability, the Cronbach α and McDonald omega estimations showed adequate internal consistency reliability for APIeHLS and its dimensions, and the split-half reliability yielded similarly strong coefficients, with values all above 0.700.

The AIPeHLS was found to be a reliable and valid instrument for assessing eHL in adult inpatients. The inclusion of adult inpatients from various specialties during the development and validation process ensures that the tool is grounded in real-world experiences and needs. Its comprehensive framework and strong psychometric properties make it an effective tool for health care providers to understand patients’ digital health competencies and tailor interventions accordingly. For researchers, our findings provided opportunities to explore the relationship between eHL and health outcomes. Notably, this study represents a significant advancement in eHL measurement by involving the latest technology usage behaviors, offering valuable insights into the development of more effective eHealth interventions and policies.

It is noted that this study has some limitations to be addressed in future research. First, criterion-related validity was evaluated using the Chinese version of the eHEALS, which is often regarded as the gold standard for measuring eHL; however, its ability to accurately reflect the true level of eHL within the target population was significantly constrained by its overly concise and outdated items. Second, the representativeness of results might be limited due to the study sample selected exclusively from a single clinical institution in China. Third, the absence of test-retest reliability due to the relatively short periods of hospital stays may affect the stability of the instrument over time. Accordingly, multicenter studies across diverse populations and time points are expected to further evaluate the measurement invariance and longitudinal reliability of AIPeHLS. Despite these limitations, this study addressed a critical gap in the literature as no prior measure has specifically focused on eHL assessments among inpatient populations who must navigate increasingly complex eHealth challenges. This work represented an innovative advancement in understanding and measuring eHL, particularly by integrating the evolving Web3.0 context and the rapidly advancing eHealth technologies worldwide. Moving forward, the relationships between eHL, as measured by the AIPeHLS, and a range of potential health-related variables could be systematically explored. Furthermore, this scale can be incorporated into studies evaluating the effectiveness of digital health interventions in inpatient settings, providing valuable insights into the impacts of eHealth tools on the health outcomes, self-management capabilities, and overall well-being of hospitalized individuals.

A psychometrically robust, multidimensional instrument termed the AIPeHLS was developed and validated in this study, comprising 44 items that comprehensively cover all 6 dimensions of the theoretically grounded Lily model of eHL. The AIPeHLS demonstrates a substantial potential to serve as a reliable and valid means of measuring eHL among adult inpatient populations in the evolving Web 3.0 context, empowering health care providers to better understand and improve eHL of inpatients. Furthermore, the deployment of the AIPeHLS may facilitate researchers, engineers, and healthcare providers in evaluating and implementing effective eHealth interventions across diverse clinical settings.