The study was developed through a 3-phase process [47]. Phase 1 involved a literature review to identify conceptual domains of AI literacy relevant to health care workers and to derive initial scale items capturing the identified domains. Phase 2 aimed to evaluate the initial scale items through expert review and pilot testing with 4 experts and 26 health care workers, including assessments of content and face validation. Phase 3 evaluated the validity and reliability of the final scale items quantitatively with 425 health care workers. This included cross-sectional survey research and statistical analysis. This study was reported in accordance with the Consensus-Based Standards for the Selection of Health Measurement Instruments (COSMIN) guidelines for studies on measurement properties [48], and the completed COSMIN checklist is provided as a supplementary file (Checklist 1).

In all stages, participants were required to meet the following criteria: (1) be currently working in a health care–related role, including clinical, allied health, administrative, or ancillary positions; (2) be able to complete an English-language questionnaire; and (3) be willing to provide informed consent. Participants were health care workers employed across acute and tertiary hospitals, specialist services, and community care settings within a major health care provider in Singapore (NHG Health). Additionally, participants needed to be capable of completing study procedures, including expert evaluation or survey participation, as required in each stage.

Participant recruitment was conducted across multiple institutions within NHG Health using a convenience sampling approach. Data collection for the study took place between March and July 2025. Participants were recruited across Phases 2 and 3 through institutional communication platforms and internal mailing channels. On accessing the survey link, participants were provided with an information sheet describing the study aims, procedures, and data protection measures before providing informed consent.

For Phase 2 (face validation), 26 health care workers were recruited from NHG Health to participate in pilot testing of the scale to assess item clarity and interpretability. Participation was voluntary, and no incentives were provided. Additionally, experts were recruited for Phase 2 (content validation), including 4 experts comprising 1 clinician, 1 AI specialist, and 2 health systems professionals with experience in health care delivery and technology implementation. Experts were independent of the study team, provided informed consent prior to participation, and received no incentives.

For Phase 3 (scale evaluation), the target sample size was determined based on recommended respondent-to-item ratios [49]. For the 28-item version 2 of the SAIL-HCW scale, a minimum ratio of 10 participants per item indicated a minimum sample size of 280, while a preferred ratio of 15 participants per item yielded an ideal target of 420.

The primary objective of this study was to develop and validate a scale to assess AI literacy among health care workers. The instrument was designed to capture key competencies related to understanding, evaluating, and applying AI in health care settings. The overall process of scale development is shown in Figure 1.

Data were analyzed in accordance with the 3 phases of scale development. Each phase involved distinct but complementary analytic approaches aligned with the objectives of item generation, scale refinement, and psychometric evaluation. The analyses for each phase are described below.

In Phase 1, domains and subdomains identified from the literature were analyzed through iterative comparison and grouping of conceptually similar elements. Redundant or overlapping domains were consolidated, and items were mapped to the corresponding domains to ensure conceptual alignment. The process was conducted independently by 2 authors (CSA and SI), with a third author (SB) consulted to resolve discrepancies where necessary. All derived domains and items were then independently reviewed by other team members who had not been involved in the initial scale development process. Each item was assessed for clarity, conceptual relevance, suitability for health care workers, and potential overlap with other items. Feedback from the review was subsequently discussed within the research team before final item refinement.

In Phase 2, content validity and face validity data were analyzed using both quantitative and qualitative approaches. For content validity, item-level content validity indices (I-CVI) were calculated as the proportion of experts rating each item as relevant, and items with I-CVI values below 1.00 were considered for removal [53]. For face validity, item-total correlations (ITC) were calculated to identify items with weak associations with their respective domain scores; items with ITC values below 0.30 were considered for removal [25]. Qualitative analysis for both content and face validity involved reviewing open-ended participant feedback to assess item clarity, interpretability, and relevance, as well as identifying common issues across responses.

Survey data from Phase 3 were analyzed using SPSS (version 26.0; IBM Corp) and AMOS (version 24.0; IBM Corp). The dataset was first reviewed for completeness and response quality [54], and descriptive statistics were used to summarize the sample. Prior to factor analysis, the distributional properties of SAIL-HCW item scores were examined. Univariate outliers were screened using z scores, with values outside 3 considered extreme [55]. Skewness and kurtosis were assessed for univariate normality, with values within 2 considered acceptable [55]. Multivariate outliers were assessed using Mahalanobis distance with reference to the chi-square distribution (P<.001), and multivariate normality was evaluated using Mardia coefficients for skewness and kurtosis.

Given the logistical challenges associated with conducting research in busy health care settings [52], data were collected at a single time point to minimize respondent burden. A split-sample approach was used to allow independent testing of the factor structure. The dataset was randomly divided into 2 comparable subsamples for separate exploratory and confirmatory factor analyses. Sample 1 was used for exploratory factor analysis (EFA) and Sample 2 for confirmatory factor analysis (CFA). Equivalence between the 2 subsamples was assessed prior to analysis.

EFA was conducted to examine the underlying factor structure and how items clustered without imposing a predefined model, as well as to identify weak or cross-loading items. The suitability of the data for EFA was assessed using the Bartlett test of sphericity and the Kaiser-Meyer-Olkin (KMO) measure. A significant Bartlett test indicates sufficient interitem correlations for factor analysis, while KMO values above 0.70 indicate adequate sampling [56]. EFA was performed using principal axis factoring, with direct oblimin rotation applied to allow for correlations between latent factors. The number of factors to retain was determined using multiple criteria. Eigenvalues greater than 1.0 were used as an initial guide [57], while the scree plot was inspected to identify the point at which the slope of eigenvalues leveled off, indicating diminishing returns from additional factors [58]. Parallel analysis was also used to compare observed eigenvalues with those generated from random data; factors were retained when observed eigenvalues exceeded the corresponding random values [59]. At the item level, criteria for retention included acceptable primary factor loadings and communalities, with items expected to demonstrate loadings on a single factor and communalities of at least 0.40. Secondary loadings ≥0.20 were used as an indicator of potential cross-loading and conceptual overlap across constructs [55].

CFA was subsequently conducted to evaluate the dimensional structure of the SAIL-HCW by comparing empirically and theoretically informed models. Although EFA findings suggested a domain-general factor, the scale had been conceptually developed around 7 theoretically driven domains. To examine these competing representations of AI literacy, 4 alternative measurement models were tested. Model 1 specified a one-factor structure, in which all items loaded onto a single general AI literacy factor. Models 2‐4 reflected the assumption that AI literacy comprises 7 related but distinct domains. Model 2 specified a correlated 7-factor structure, allowing the domains to be associated while remaining conceptually distinct. Model 3 specified a higher-order structure, in which the 7 domains were represented as manifestations of a broader overarching AI literacy construct. Model 4 specified a bifactor structure, in which each item loaded simultaneously onto a general factor, representing AI literacy, and onto its corresponding domain-specific factor, representing more specific competencies. This approach allows evaluation of whether the scale functions as a unidimensional construct, a multidimensional construct, or a combination of both, and provides a more comprehensive assessment of the underlying structure [60]. All models were estimated at the item level without parceling. All analyses used weighted least squares mean and variance adjusted (WLSMV) estimation to account for the ordinal response format [61].

Model fit was assessed using the comparative fit index (CFI) and Tucker-Lewis index (TLI), with values ≥0.90 indicating acceptable fit [62]; the root-mean-square error of approximation (RMSEA) with 90% CI, and the standardized root-mean-square residual (SRMR), with values ≤0.08 indicating acceptable fit [61,63]. Akaike Information Criterion (AIC) values were used to support model comparison, with lower values indicating better relative model fit and greater parsimony [64]. Modification indices were reviewed to identify potential areas of model misfit and to determine whether theoretically justifiable modification could improve model fit. Standardized factor loadings greater than 0.50 were taken as indicative of good item performance [63]. After the factor structure was established, analyses were conducted using the full sample to evaluate item performance, reliability, and validity.

Internal consistency was assessed using Cronbach alpha (α) and McDonald omega (ω) coefficients, which estimate the extent to which items within the scale measure the same underlying construct [65]. Values of ≥0.60 were considered acceptable, and values ≥0.70 were considered good [55]. Item analysis was conducted to evaluate the performance of individual items in terms of their ability to discriminate between respondents and their contribution to the overall scale. Item discrimination was assessed using the 27% upper-lower group method. Total scores were ranked, and respondents in the highest and lowest 27% of the distribution (n=115 in each group) were compared using independent samples t tests [47]. Significant differences between groups indicated that items effectively distinguished between higher and lower levels of AI literacy. Corrected item-total correlations (CITCs) were calculated to examine the extent to which each item aligned with the overall scale. Values ≥0.30 were considered acceptable, indicating that items made a meaningful contribution to the construct being measured [55]. Interitem correlations were also examined to assess redundancy and the degree of overlap between items. Values below 0.80 were considered acceptable, indicating adequate discriminant performance between items and no evidence of redundancy [66].

Finally, construct validity was assessed using a known-group approach [47]. SAIL-HCW scores were compared across groups expected to differ in AI literacy based on prior AI-related training, frequency of AI use, and self-rated AI literacy, with higher scores indicating higher levels of AI literacy [25,26,50,51]. For analytic purposes, formal AI-related training was dichotomized into no training versus some or extensive training. Frequency of AI use was grouped as low (never or rarely), moderate (monthly or weekly), and high (daily). Self-rated AI literacy was grouped as low (very low or low), moderate, and high (high or very high). Group differences were examined using independent t-tests or one-way analysis of variance, as appropriate, with statistically significant differences indicating evidence of construct validity.

This study was approved by the Nanyang Technological University Institutional Review Board (IRB-2025‐069). All participants provided informed consent prior to participation. Participation was voluntary, and participants were informed that they could withdraw from the study at any time without penalty by exiting the survey before submission, without providing any explanation. No compensation was provided for participation. To acknowledge participants’ time, respondents were offered the option to enter a prize draw to win one of 20 Singapore $100 vouchers (approximately US $78; SGD $1=US $0.78 as of July 15, 2025). Contact details collected for this purpose were stored separately from survey data and were deleted after completion of the draw. Winners were selected using a random number generator (GraphPad; GraphPad Software LLC), with generated numbers corresponding to participant identifiers. There were no penalties or negative consequences to participants’ employment, benefits, or entitlements if they chose not to participate or withdrew from the study, and any data from participants who withdrew were not processed or analyzed. All data were anonymized and stored securely, with access restricted to authorized members of the research team. Any potentially identifiable information collected during the study was removed after completion of data collection.

Quantitatively, content validation identified 19 items with I-CVI values below the predefined threshold, while face validation identified one item with a low ITC. These items were removed from the scale.

Qualitative feedback from experts was generally positive. Experts supported the conceptual appropriateness and comprehensiveness of the 7-domain structure, with no additional domains proposed. They also highlighted the importance of ensuring balanced item coverage across domains. Similarly, face validation feedback from health care workers indicated that participants’ interpretations were generally consistent with the intended domains, suggesting that the domain structure was intelligible to the target population. Both experts and health care workers also agreed that the 6-point Likert scale was appropriate.

In addition, qualitative feedback identified areas for refinement. Both experts and health care workers noted that 14 items overlapped conceptually with other retained items. These were not retained as separate items; instead, their content was integrated into existing items to reduce redundancy and improve clarity. Health care workers also reported that 4 items containing technical terminology were difficult to interpret. These items were retained, with minor wording revisions made to improve comprehensibility without item exclusion. In response to expert feedback on the need for balanced coverage, 6 new items were added to improve representation across the 7 domains. This process resulted in a version 2 of the SAIL-HCW scale (28-item; see Table 2).

Participants with prior AI-related training scored higher on the SAIL-HCW than those without training. Scores also increased with frequency of AI use, with higher scores observed among participants reporting more frequent use. A similar gradient was observed for self-rated AI literacy, with progressively higher SAIL-HCW scores across increasing self-assessed literacy levels. All group differences were statistically significant (Table 7). These findings were consistent with theoretical expectations and support the construct validity of the SAIL-HCW.

This study developed and evaluated the SAIL-HCW, a brief instrument intended to assess AI literacy among health care professionals (see Multimedia Appendix 4). The findings provide preliminary evidence supporting the scale’s structural validity, internal consistency, item discrimination, and construct validity. A notable finding concerned the underlying structure of AI literacy within the sample. Although AI literacy was conceptualized through 7 theoretically distinct domains, the analyses consistently indicated the presence of a dominant general factor. Among the competing confirmatory models examined, the bifactor model provided the best fit to the data, suggesting that AI literacy among health care workers may be understood as an overarching capability that also includes distinguishable domain-related components.

The conceptualization of AI literacy as a multidimensional construct is consistent with previous literature. Existing frameworks commonly describe AI literacy as extending beyond technical knowledge to include competencies related to understanding data, evaluating AI-generated outputs, ethical awareness, practical application, and adaptation to evolving technologies [10,16,21,22,28,35-44]. Drawing on this literature, this study identified 7 domains considered relevant to health care practice, including conceptual understanding of AI systems, data fluency, evaluation of AI outputs, practical application, ethics and regulation, organizational and sociotechnical awareness, and continuous learning. These domains informed the development of the SAIL-HCW and provided a theoretically grounded framework for operationalizing AI literacy within health care settings.

While the 7-domain framework was conceptually useful, the findings suggest that health care workers may not necessarily experience these competencies as fully distinct in practice. Responses across domains were strongly influenced by a common underlying factor, despite evidence of domain-related variation. This pattern may reflect the nature of health care work, where AI-related tasks often involve multiple competencies operating simultaneously. Interpreting AI outputs, for example, may require conceptual understanding, awareness of data quality, ethical judgment, and practical knowledge of clinical workflows at the same time [15,45]. From this perspective, AI literacy may function less as a set of isolated competencies and more as an integrated professional capability shaped by the realities of clinical and organizational practice.

The bifactor findings help clarify this relationship between multidimensionality and unidimensionality. The bifactor model showed better fit than the one-factor, correlated-factor, and higher-order models, indicating that responses to SAIL-HCW items were influenced both by a general AI literacy factor and by smaller domain-specific factors. Most items loaded more strongly onto the general factor than onto their respective domains, suggesting that respondents tended to approach the items through an overall perception of AI literacy rather than through clearly differentiated competency areas. At the same time, the domain-specific factors retained conceptual relevance, indicating that the 7 domains still captured meaningful aspects of AI literacy beyond the general construct [60].

These findings suggest that AI literacy may be conceptualized as multidimensional while remaining sufficiently unidimensional for practical score interpretation [75]. This distinction may be important for both assessment and educational use. The general factor supports the use of an overall SAIL-HCW score as an indicator of perceived AI literacy, which may be useful in workforce assessment and evaluation contexts. At the same time, the 7 domains may still provide useful information regarding relative areas of strength and limitations. For example, health care workers may report greater confidence in practical AI use while demonstrating less familiarity with ethical, regulatory, or data-related aspects of AI. In this sense, the domains may function more meaningfully as educational and developmental categories than as fully independent psychometric subscales.

The findings also contribute to current discussions regarding the measurement of AI literacy in health care settings. Existing instruments have largely focused on students, trainees, or general populations [37-39,41,43]. Although these measures have contributed to the early development of the field, they may not fully reflect the applied and context-dependent nature of AI use in health care practice. Health care workers are required not only to understand AI systems, but also to evaluate outputs critically, recognize uncertainty and limitations, navigate ethical and organizational considerations, and integrate AI into existing clinical and operational workflows [8-11]. The inclusion of domains such as ethics and regulation, sociotechnical awareness, and continuous learning reflects these broader professional expectations and extends beyond the narrower technical focus commonly reported in earlier instruments.

The continuous learning domain may be particularly relevant within health care contexts. Unlike some forms of professional knowledge that remain relatively stable over time, AI technologies continue to evolve rapidly, often more quickly than formal training programs or institutional guidance [17]. AI literacy may therefore involve not only existing knowledge and skills, but also the capacity to recognize limitations in one’s understanding and adapt to emerging technologies and changing practice environments. The inclusion of continuous learning within the SAIL-HCW reflects this more adaptive view of AI literacy and aligns with wider discussions regarding lifelong learning in digitally enabled health care systems.

The reliability and item-level findings provide additional support for the scale. Internal consistency coefficients for the total scale were high, while domain-level coefficients were acceptable given the brief nature of the subscales [55]. All retained items demonstrated meaningful CITCs and differentiated between respondents with higher and lower levels of AI literacy. Known-group comparisons also showed that SAIL-HCW scores varied according to prior AI-related training, frequency of AI use, and self-rated AI literacy. These patterns were consistent with theoretical expectations and provide additional support for the construct validity of the scale [25,26,50,51].

Overall, the findings suggest that AI literacy among health care workers may be understood as an integrated yet multifaceted capability. Although the 7 domains remain theoretically and practically relevant, responses to the SAIL-HCW were primarily shaped by a dominant general factor. The bifactor structure therefore provides one possible way of understanding AI literacy in health care settings, allowing both overall score interpretation and consideration of domain-related variation.

This study reports the development and validation of the SAIL-HCW, an instrument designed to assess AI literacy among health care workers. The scale offers an initial structured measure of a construct that is increasingly relevant as AI becomes more embedded in health care delivery and organization. Findings from the psychometric evaluation suggest that AI literacy in this sample can be represented as a general construct, with 7 underlying domains capturing distinct but related aspects of competence. The scale demonstrated acceptable levels of internal consistency and showed expected patterns of item performance and group differences, supporting its use as a preliminary measure for both research and educational purposes. The SAIL-HCW may be useful for identifying variation in AI literacy within health care workforces and for informing the design of training and educational activities. At the same time, further work is likely needed to examine how the scale performs across different health care settings, professional groups, and levels of exposure to AI technologies. As AI continues to evolve within health care systems, tools that support the assessment of workforce capability may help to clarify current levels of understanding and highlight areas where further development is needed.