Introduction
The global realm of digital health apps has grown dramatically. This trend has been particularly notable since the onset of the COVID-19 pandemic in early 2020 [1,2]. As of the first quarter of 2021, more than 53,000 apps were available on the Google Play Store. This represents a notable increase of over 25% compared to about 42,000 apps on the same platform during the same period of the previous year [3]. These apps offer a feasible alternative to face-to-face communication between health care professionals and patients, assist health care professionals in diagnosing and managing various medical conditions by providing quick access to health data, disseminate health-promoting knowledge, and facilitate patients’ self-management, thus improving the efficiency of the health care system worldwide [4,5].
As digital health apps gain increasing popularity, the market is saturated with a diverse array of such apps, each varying in quality and functionality. Consequently, users frequently encounter difficulties in selecting the appropriate and truly useful apps that can enhance their health outcomes. A rigorous and standardized app rating system, implemented before the release of these apps to the major stores, empowers users to make well-informed decisions and fosters the healthy development of the app market [6]. However, the star rating and user reviews provided on the App Store are subjective and cannot accurately reflect the true usefulness and effectiveness of a digital health app [7]. In contrast, usability, defined as the extent to which users can use an app to achieve specific objectives with efficiency, satisfaction, and effectiveness in a specified usage context, is crucial to reflect the quality and efficacy of digital health apps [8,9]. Therefore, a rigorous and validated usability evaluation tool is urgently warranted to produce objective usability results for digital health apps before their release. This would enable consumers and researchers to promptly and efficiently select reliable apps.
Several questionnaires have been developed previously for usability testing, such as the System Usability Scale (SUS) [10], the Post-Study System Usability Questionnaire (PSSUQ) [11], the Software Usability Measurement Inventory [12], and the Computer System Usability Questionnaire (CSUQ) [13]. However, these questionnaires were originally designed with a focus on general information technology systems without considering the unique characteristics of digital health apps. These include the specialized content tailored to health needs, users’ high expectations regarding the accuracy of the information and functions, and the involvement of both service consumers (ie, patients who rely on the personalized chronic disease management apps to manage chronic diseases, access medical and health care information, and communicate with health care providers) and professional health care providers (ie, medical and nursing staffs involved in the personalized chronic disease management apps to monitor patients’ health status, provide medical advice, and coordinate treatment plans). Consequently, these usability evaluation tools prove challenging to reliably identify the specific problems that may arise when using digital health apps [14].
To bridge this gap, Yen et al [15] developed the Health Information Technology Usability Evaluation Scale (Health-ITUES) based on Bidshift, a web-based communication system for scheduling nursing staff to improve the efficiency and effectiveness of the staffing and scheduling process. Bidshift allows nurse managers to announce open shifts throughout the organization and staff nurses to request shifts. Consequently, the Health-ITUES was developed to assess nurses’ usability perceptions regarding the Bidshift system for shift requests at the task, individual, and organizational levels. It has been increasingly used as a validated tool to specifically evaluate the usability of digital health apps by clearly considering tasks [16]. The Health-ITUES also supports the customization at the item level to align with the specific tasks and expectations of the health systems while retaining comparability at the construct level [17]. The original English version of the Health-ITUES has been validated through exploratory factor analysis (EFA) and confirmatory factor analysis (CFA) among nurses [16] and community-dwelling adults with HIV [18]. Although a validated Korean version of Health-ITUES already exists [19], a Chinese version is currently unavailable. This absence creates a significant gap, especially considering the language, cultural norms, and the distinct regulatory framework and professional practices inherent in the Chinese health care system.
Therefore, this study aimed to translate and culturally adapt the Health-ITUES into Chinese, revise its customized parts to cater to both service consumers and professional health care providers, and evaluate its reliability and validity in the Chinese context.
Methods
Study Design
This study followed the guideline for the process of cross-cultural adaptation of self-report measures by Beaton et al [20]. The guideline offers a well-established and systematic framework to ensure rigor and validity of the cross-cultural adaptation of the Health-ITUES, minimizing potential biases and errors that might arise during the translation and adaptation processes. It also enables comparability with other studies that have followed the same or similar procedures and facilitates a more meaningful synthesis of research findings within the field. Before initiating the translation process, we obtained permission from the original author of the Health-ITUES via email to translate it into Chinese (refer to Figure S1 in Multimedia Appendix 1). This paper was reported in accordance with the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist (Multimedia Appendix 2).
The Original Health-ITUES
The Health-ITUES was originally designed to measure the usability of a web-based communication system for scheduling nursing staff. Derived from the health information technology usability evaluation model, a theoretical framework to guide usability evaluations of digital health technologies [21], the Health-ITUES has recently been increasingly used to accurately assess the usability of digital health apps [18,19]. The tool allows for customization of the items to match the specific tasks and expectations of the health systems. It comprises 20 items from 4D: quality of work life (3 items), perceived usefulness (9 items), perceived ease of use (5 items), and user control (3 items). Each item is rated from 1 (strongly disagree) to 5 (strongly agree) on a 5-point Likert scale [15]. The total scores range from 20 to 100, with higher scores indicating better perceived usability. The English version of the Health-ITUES demonstrated satisfactory reliability and validity, with Cronbach α coefficients and criterion validity indexes of 0.85-0.92 and 0.46-0.70, respectively [16].
Translation and Cross-Cultural Adaptation of the Health-ITUES
Forward Translation and Synthesis of the Forward Translations
In total, 2 bilingual native Chinese speakers who were proficient in English and had passed the College English Test Band Six produced 2 forward Chinese translations of the Health-ITUES independently (T1 and T2). One of the translators was familiar with the Health-ITUES contents, while the other was unaware of the concepts being quantified. Subsequently, the 2 translations were meticulously reviewed for any ambiguity until a consensus was reached. In the process, we also invited a third translator to resolve disagreements. Through iterative comparison and refinement, a synthesized Chinese version, T3, was achieved.
Back Translation
The synthesized Chinese version T3 was independently back-translated into English versions (BT1 and BT2) by another 2 experienced translators who were native English speakers and had a good command of Chinese. Both of the back translators were blinded to the original Health-ITUES. The research team then compared the 2 back translations, analyzed the similarities and differences between them, and provided feedback to the original author of the Health-ITUES for verification. The basic information of the translators was summarized in Table S1 in Multimedia Appendix 1.
Cross-Cultural Adaption
All translated versions of the Health-ITUES (including the 2 forward translations, the synthesized version, and the 2 back translations) were thoroughly discussed and evaluated by a panel of 6 experts with varied research fields, encompassing older welfare technology, clinical nursing, nursing information, Chinese and American culture, and data science and engineering (refer to detailed information in Table S2 in Multimedia Appendix 1). The experts were invited to provide feedback on the accuracy of translation and professional terminology, readability, seamless integration with the language of the information system, comprehensibility for nonprofessionals, alignment with clinical practice, cultural appropriateness, and item retention. Through the adoption of a self-evaluation method (Table S3 in Multimedia Appendix 1), the authority coefficient (Cr) was determined by considering the familiarity coefficient (Cs) and the judgment coefficient (Ca). The formula for calculating the Cr is Cr = (Cs + Ca)/2 [22].
Based on their professional theoretical knowledge and practical experience, the experts evaluated each item independently on semantic equivalence, conceptual equivalence, experiential equivalence, and idiomatic equivalence. Any items with ambiguity were reworded until all expert queries were addressed, eventually resulting in the final Chinese version of the Health-ITUES.
Validation of the Health-ITUES
Overview
We used the SMART system (known in Chinese as Aifuxing App), developed in the early stage, as a digital health App for the Health-ITUES validation. In short, the SMART system was primarily designed to facilitate personalized integrated home-based care for older people [23]. Considering that older individuals and professional care providers are the main users of the app, this study aimed to validate the effectiveness of the Chinese version of the Health-ITUES among both older people and professional health care providers.
Customization of the Care Receiver and Professional Health Care Provider Versions of the Health-ITUES
Based on the overall objectives and functional components of the SMART system, the research team engaged in multiple rounds of discussions to refine the customized components within the Chinese version of the Health-ITUES, formulating the initial care receiver (ie, Health Information Technology Usability Evaluation Scale—care receiver version [Health-ITUES-R]) and professional health care provider versions of the Health-ITUES (ie, Health Information Technology Usability Evaluation Scale—professional health care provider version [Health-ITUES-P]). Subsequently, a panel of 6 experts from pertinent fields (refer to specific details in Table S4 in Multimedia Appendix 1) were tasked with reviewing the 2 initial customized scales and assigning ratings for the correlation between each item and the corresponding dimension on a 4-level scale: 1=uncorrelated, 2=weakly correlated, 3=strongly correlated, and 4=highly correlated. The research team then iteratively modified the items according to expert suggestions until a consensus was reached among all experts on the finalized Health-ITUES-R and Health-ITUES-P.
Study Participants
The validation test was conducted in a geriatric ward of a comprehensive hospital in Beijing, China from December 2020 to February 2021. A total of 3 primary factors underlie our choice of this research time frame. First, during the COVID-19 pandemic, the hospital adopted a closed-door management strategy that strictly restricted inpatient numbers and prohibited nonhospital staff from entering. This situation persisted until December 2020, when the improving pandemic conditions enabled patient admissions and researchers’ entry for recruitment activities. Concurrently, a significant increase in patient arrivals guaranteed an adequate sample size. Second, the research period extended just before the Spring Festival, a major traditional Chinese festival, which further enhanced the robustness of the sample size to generate reliable results. In addition, the surge in digital health app use during the COVID-19 pandemic allowed us to assess usability in a relevant, real-world health care environment. Older adults were included consecutively if they (1) were aged 60 or older, (2) possessed normal communication and interaction abilities, (3) obtained at least a primary school education, (4) had an android-based smartphone for internet access, and (5) expressed willingness to participate. Older individuals with dementia or other mental illness were excluded to ensure comprehension of the scale items. To validate the Health-ITUES among professional health care providers, nurses, considered as the primary professional health care providers, were included. In-service nurses holding nurse qualification certificates were enrolled in the study if they had an android-based smartphone and were willing to participate.
Ethical Considerations
The study followed the Declaration of Helsinki and received approval from the institutional review committee of the Capital Medical University (approval number 2015SY49U). Potential participants were thoroughly informed of the study’s objectives, methods, procedures, and the data that would be collected, as well as their right to discontinue their participation at any time without facing any adverse impacts. Only those who provided written informed consent were enrolled. In addition, to safeguard participants’ privacy, personal identifiers were stored in password-protected files, and only deidentified data were used for analysis and reporting purposes. In recognition of their contribution, each participant was rewarded with a small token valued at ￥20 (about US $3). The manuscript and supplementary materials were meticulously designed to exclude any information that could disclose the identities of individual participants.
Instruments
General Information Collection
Demographic information including age, gender, education, and monthly income was collected from both older individuals and nurses. Besides, we gathered data on nurses’ professional titles and years of employment. To measure participants’ usage frequency of common functions on mobile phones, a mobile phone usage experience questionnaire was used. This questionnaire, derived from the questionnaire on computer experience, consists of 8 items, each rated on a 5-point Likert scale ranging from 1 (never) to 5 (very frequently). The total scores range from 8-40, with 8-16, 17-32, and 33-49 indicating low, moderate, and abundant mobile phone usage experience, respectively. The questionnaire exhibited satisfactory reliability, with a Cronbach α coefficient of 0.922 [24].
The Health-ITUES-R and Health-ITUES-P
The finalized Health-ITUES-C and Health-ITUES-P adapted from the Chinese version of the Health-ITUES were used to collect the perceived usability of the SMART system among older individuals and nurses, respectively. Respondents rate these items on a 5-point Likert scale from 1 (completely disagree) to 5 (completely agree). Total scores for both versions are calculated by summing the scores of each item, with higher scores reflecting better-perceived usability.
Patient Acceptance Questionnaire for the Mobile Health App
The patient acceptance questionnaire for the mobile health app was used as a reference standard to evaluate the criterion validity of the Health-ITUES. This questionnaire, designed to evaluate patients’ acceptance of mobile medical products, consists of 32 items covering 6D: usefulness, ease of use, trust, usage attitude, system interface, and usage tendency. Respondents provide ratings for each item on a 5-point Likert scale ranging from 1 (strongly disagree) to 5 (strongly agree), resulting in a total score of 160. The questionnaire had satisfactory reliability and validity, with the scale-content validity index (S-CVI), Cronbach α coefficient, and split-half reliability of 0.97, 0.96, and 0.99, respectively [25]. Given its focus on users of medical apps, the questionnaire was only administered to older people.
Data Collection Procedures
After providing a comprehensive explanation of the study’s purpose, significance, and procedures, the research team assisted the participants in downloading and installing the SMART system and completing registration and login. Training materials, including instructional videos and user manuals, were made available to the participants until they felt confident in using the app. Subsequently, the participants were required to use the SMART system independently for 24 hours to complete the assigned tasks before filling out the Health-ITUES-R, Health-ITUES-P, and patient acceptance questionnaire for the mobile health app as appropriate. During the 24-hour app-use period, the enrolled older participants and nurses were instructed to complete the assigned tasks independently, without seeking assistance or discussing related content with others. Continuous monitoring was conducted by the research team and nurses from the department who were not enrolled as participants. The detailed tasks assigned to older adults and nurses are listed in Table S5 in Multimedia Appendix 1.
Sample Size Calculation
To achieve adequate statistical power, the sample size should be 5-10 times the number of items [26]. With a total of 20 items in the Health-ITUES, the study necessitated a minimum of 100 participants. Taking a 20% dropout rate into consideration, at least 110 older individuals and nurses were needed for the study. The anticipated dropout rate of 20% was based on prior studies investigating app usability [27,28].
Statistical Analysis
Continuous data was tested for normal distribution by using the 1-sample Kolmogorov-Smirnov test and expressed as mean SD or medians with IQR as appropriate. For the between-group comparison, the student t test was used for continuous variables with normal distribution, while the Mann-Whitney U test was used for nonnormally distributed continuous data. Categorical variables were expressed as frequencies or proportions (%) and comparisons were conducted using chi-square or Fisher exact test as appropriate.
The content validity of the finalized Health-ITUES-R and Health-ITUES-P was assessed by the item-level content validity index (I-CVI) and S-CVI based on expert ratings. I-CVI is the ratio of the experts ranking the item for 3 or 4 scores, and the S-CVI is the average value of all the I-CVI scores [29]. A scale with an I-CVI of > 0.78 and a S-CVI of ≥ 0.90 is considered satisfactory [30].
The internal consistency reliability was determined by Cronbach α, McDonald ω, and corrected item-total correlation coefficient (CITC). Values of Cronbach α and McDonald’s ω ≥ 0.70 are considered adequate, while a value of CITC of <0.30 indicates a low correlation [31,32]. While the test-retest reliability could assess measurement consistency under consistent conditions, the fluctuating nature of users’ perceived usability of the digital health apps over time makes this indicator unsuitable [33]. Furthermore, due to inherent variations in user perceptions of the app’s usability, interrater reliability was not examined in the study.
CFA with maximum likelihood estimation was performed to explore the structure validity. The analysis provided standardized factor loading to estimate the relationship strength between items and dimensions [34], together with model fit indices, including χ2/df, root-mean-square error of approximation (RMSEA), root-mean-square residual (RMR), standardized root-mean-square residual (SRMR), parsimonious goodness-of-fit index (PGFI), parsimonious normed fit index (PNFI), and parsimonious comparative fit index (PCFI). Acceptable structure validity was evaluated using recommended cut-offs characterized as standardized factor loading of ≥0.60, χ2/df of <3, RMSEA of ≤0.10, RMR of ≤0.05, SRMR of ≤0.80, PGFI of ≥0.50, PNFI of ≥0.50, and PCFI of ≥0.50 [35].
To determine the convergent validity, the composite reliability (CR) and average variance extracted (AVE) were calculated through the Fornell and Larcker approach [36] with a CR ≥ 0.70 and AVE ≥ 0.50 indicating satisfactory convergent validity. The square root of the AVE exceeding each of its correlations with other dimensions indicates appropriate discriminant validity [37]. The discriminant validity was also tested by the Heterotrait-Monotrait ratio (HTMT), where a value <0.85 is acceptable [38]. In addition, the criterion validity between the Health-ITUES and patient acceptance questionnaire for mobile health apps was analyzed through Pearson correlation analysis, with correlation values of >0.50 deemed adequate [39].
Statistical analyses were performed using AMOS version 26.0 (IBM Corp) for CFA and SPSS version 26.0 (SPSS Inc) for the remaining analyses. A 2-sided P value of <.05 was considered statistically significant.
Discussion
Principal Findings
Based on the original Health-ITUES, we meticulously translated and culturally adapted it to develop the Chinese version of the Health-ITUES. The validation test conducted among the main users of the SMART system (older people and nurses) confirmed satisfactory reliability and validity of the Chinese version of the Health-ITUES in evaluating the usability of digital health apps. To our understanding, this represents the first exploration of a valid usability evaluation instrument specifically designed for digital health apps considering both care receivers and professional health care providers in China, which can provide evidence supporting the use of the Chinese version of the Health-ITUES as a validated tool for evaluating the usability of digital health apps.
Following the guidelines for the process of cross-cultural adaptation of self-report measures [20], we carefully selected appropriate translators for both forward and back translations of the Health-ITUES and determined the Chinese version through numerous rounds of discussions within the research team [40]. Simultaneously, our iterative modification process under consultations with the expert panel until obtaining verification of the original author, enabled us to adjust the dimensions and items from a professional perspective and ensure that the original meanings of the Health-ITUES items were retained, thereby improving the effectiveness and practicality of the Chinese version of the Health-ITUES [41].
Furthermore, the customized Health-ITUES-R and Health-ITUES-P were validated as effective tools with good content validity, internal consistency reliability, and discriminant validity in measuring the usability of the SMART system. The CFA results also indicated an adequate structure validity, except for a slightly higher RMSEA value of the Health-ITUES-P. It is worth noting that the RMSEA value is calculated based on non-centrality parameters for representing the absolute measure of fit, and its calculation heavily depends on the sample size [42,43]. Models with smaller sample sizes were generally believed with the potential to artificially large RMSEA values, which can explain the overestimated RMSEA value in our validation test to some extent [44]. Regarding the convergent validity, the slightly lower AVE value for the second dimension in the Health-ITUES-R (0.478) could be attributed to the limited understanding and short usage time of the SMART system among older individuals. This may result in less precise responses to the 9 questions in this dimension and a lower AVE value [45]. In addition, the Health-ITUES-R exhibited high criterion validity compared to the patient acceptance questionnaire for mobile health apps.
To the best of our knowledge, our study, for the first time, formulated the Chinese version of the Health-ITUES and validated its utility for evaluating the usability of digital health apps in the Chinese context by considering both care receivers and professional health care providers [8]. In addition to the commonly used validation measures such as content validity, internal consistency reliability, structure validity, and criterion validity, we also examined the convergent and discriminant validity for a comprehensive validation of the Chinese version of the Health-ITUES. Given its strong psychometric properties, we postulate that the Chinese version of the Health-ITUES can serve as a valuable instrument in evaluating the usability of digital health apps for both professional health care providers and receivers.
Limitations
There are several limitations in this study. First, the validation of the Chinese version of the Health-ITUES relied on the SMART system, a platform designed for personalized integrated home-based older care, while the Health-ITUES applies to all types of digital health apps. Consequently, it cannot be ruled out that the validation results could have differed with another app. Second, there may be a selection bias in the sample selection, since the validation tests were conducted in the geriatric ward of a comprehensive hospital, where individuals tended to spend more time using digital health apps to manage their health status compared to the general population [46]. Furthermore, our validation study was constrained by relatively small sample sizes and inadequate usage time of the SMART system due to the COVID-19 pandemic. The generalizability of the findings should be approached with caution. Further research with a larger sample and adequate usage time of the digital health apps is needed.
Conclusions
This study formulated a Chinese version of the Health-ITUES with satisfactory reliability and validity in evaluating the usability of the digital health apps for both care receivers and professional health care providers. The Chinese version of the Health-ITUES can serve as a valuable tool to identify reliable and effective digital health apps for end users. Future research focusing on the validation of the Health-ITUES in diverse cultural contexts and settings is crucial for enhancing its applicability and effectiveness across different populations.