Measuring the Digital Skills of Catalan Health Care Professionals as a Key Step Toward a Strategic Training Plan: Digital Competence Test Validation Study

Background: Despite Catalonia being an advanced region in terms of digital health adoption, the “Forum for Professional Dialogue” identified the need to improve information and communication technology (ICT) competences as one of the present and future challenges for health care professionals (HPs). Objective: We aimed to validate the digital competence test developed ad hoc for this study and to measure the digital competence level of Catalan HPs to establish their current level as the baseline for designing a strategic training plan. Methods: An exploratory observational study was conducted based on a voluntary survey where sociodemographic, professional and digital tool knowledge, digital tool use, and training needs data were collected and


Introduction
Background Digital health (eHealth) is changing the way prevention, diagnosis, treatment, and health monitoring are provided to patients [1,2], while allowing universal access to equal, qualified, and cost-effective health care [3][4][5]. However, unlocking the full potential of eHealth is only possible when all actors (health care professionals [HPs], patients, managers, and policy makers) are committed to accepting and adopting information and communication technologies (ICTs) as a different way of providing or receiving care. Embracing the digital culture and developing professionals' digital skills or competences to support digital transformation in the health care sector is fundamental to achieving this objective [6], because poor digital health competence is a common perceived barrier to the implementation of eHealth services [7][8][9][10][11].
Catalonia (Northeast Spain) is one of the most advanced regions in terms of digital health adoption across Europe [8,12,13]. Despite this, in 2018, the "Forum for Professional Dialogue" identified "the need to improve competences in ICTs to advance in their use and in the design of remote healthcare services" as one of the 17 main present and future challenges for HPs [14]. The "Digital Skills for HPs (COMPDIG-Salut)" project arose with the aim of meeting this challenge by addressing three objectives: (1) defining a specific digital competence framework for HPs; (2) creating a specific evaluation and accreditation model for HPs; and (3) drawing up actions to train and qualify HPs in digital competences. Therefore, knowing the current digital competence level of Catalan HPs is an essential first step upon which to build and address all COMPDIG-Salut project goals [15,16].
Despite the availability of a wide variety of free self-efficacy, knowledge-based, and performance-based digital competence assessments [16][17][18][19][20][21][22], these instruments were found to be too long, unvalidated, or too specific to measure HPs' digital literacy levels. In 2009, the Government of Catalonia launched the Accreditation of Competence in ICTs (ACTIC) certificate with the aim of assessing citizens' digital competence [23] on the basis of 3 levels: ACTIC 1-basic, ACTIC 2-intermediate, and ACTIC 3-advanced. The competences evaluated by this accreditation have been updated over the years and are currently aligned with the European Digital Competence Framework [24]. Currently, the attainment of the ACTIC 2-intermediate level certificate is a way of improving the professional development of HPs and the employability of graduates. However, the ACTIC intermediate certificate requires too much time and too many resources to complete; therefore, it was deemed unsuitable for the purposes of our research. Therefore, to assess the digital competences of HPs working in Catalonia, a digital competence test based on the ACTIC 2-intermediate certificate was developed ad hoc.

Objective
The objective of this work is two-fold: (1) to validate the digital competence test developed ad hoc, which combines both skills and self-assessment, and (2) to assess the current digital competence level of HPs working in Catalonia and to identify the areas needing improvement.

Study Design
An exploratory, observational study based on a web-based survey was conducted by Fundació TIC Salut i Social between June 28 and July 16, 2021, among HPs currently working in Catalonia. By law [25], the definition of HPs, who make up the study population, includes dentists, dental hygienists, dental technicians, dietitians-nutritionists, occupational therapists, nurses, opticians or optometrists, pharmacists, physicians, physiotherapists, podiatrists, speech therapists, and other health or clinical specialists such as biologists, physicists or chemists, and psychologists. According to the last available report (2017) [14], the HP population in Catalonia consisted of 121,039 professionals working for public and private health care providers. Participation was both voluntary and anonymous.
The survey was conducted, and the results were reported in accordance with "Good Practice in the conduct and reporting of survey research" where appropriate [26] and in compliance with the General Data Protection Regulation.

Survey Generation and Distribution
The survey included a section recording the participants' characteristics ("descriptive survey") and an objective digital competence test (Multimedia Appendix 1). The first section recorded demographic characteristics such as gender, age, professional profile, ownership of workplace (public or private [subsidized or nonsubsidized]), region, level of health care, professional experience, self-perception of digital competence, the use of digital tools for professional purposes, the need for training in digital tools for professional purposes, interest in receiving such training for personal purposes, and whether the participant had the ACTIC 2-intermediate certificate or an equivalent qualification.
The second section was the objective digital competence test. The ACTIC 2-intermediate level certificate served as the framework of reference to evaluate the digital competence level of HPs. The most appropriate and relevant digital competences for HPs were selected from those defined in ACTIC [27] to adapt the test to their reality and context (Table 1). Then, the indicators for each of these competences were selected to determine, in items, what aspects to evaluate. The indicators are observable characteristics and consist of specific tests, be they predefined measures or other types of qualitative information. After selecting the indicators, questions referring to them were formulated. These questions are related to the definition of observable behaviors that may be put into practice in different professional areas in the Catalan health care context.
Observable behaviors are understood to be those practices or actions carried out by HPs during their professional activities (eg, searching for clinical information in databases, remote communication and collaboration with teams or patients, and using information management and content creation tools). As it was the achievement of competences that was being evaluated, the suitability of defining evaluation scenarios allowing respondents to be faced with challenges they needed to resolve was assessed. Attempting to resolve situations that are similar to real ones and giving the best digital response to the proposed challenges allows the degree of achievement of the indicators to be evaluated more effectively. It also enables other competences to be put into practice, such as problem-solving, critical thinking, and the analysis and responsible use of ICTs.
After formulating the questions for each indicator, it was necessary to close the loop by reviewing the entire process using a methodological process whereby the specification of new elements improves upon previous ones [28,29]. Test development followed an iterative process of expert consultation, prior pilot tests, and item review. For cross-validation, 8 members of the COMPDIG-Salut project were asked to evaluate the overall proposal and specifically whether the questions contemplated the defined indicators for the corresponding competence. The experts had to answer the survey questions and suggest any changes they deemed relevant by answering an open-ended question.

Searching for, selecting, and comparing information with digital tools
Case 1: Question 1 The respondent uses advanced search parameters (language, update, publication date, region, Boolean operators, etc) using an assistant or different menus and options to optimize searches and readjust search criteria.
Case 1: Question 2 The respondent is critical of the information and recognizes the limits of the internet as a single source.

Organizing information and data with digital tools
Case 1: Question 6 The respondent structures and classifies data coherently and accessibly using generic tools (spreadsheet or database) and specific tools (bookmark manager, contact manager, expense tracker, etc) to make searching easier.

Analyzing, exploiting, and visualizing data with digital tools
Case 1: Question 5 The respondent gathers data with digital tools (forms, surveys, etc) for specific objectives.
Case 2: Question 7 The respondent uses and combines formulas and functions to perform simple operations.

Interacting and sharing information and digital content
Case 1: Question 4 The respondent acts as an example of communication for the rest of the digital community according to the context and the tool used.

Collaborating with others via digital technologies
Case 2: Question 1 The respondent identifies and uses digital tools, resources, and strategies to improve efficiency in the performance of tasks in collaboration with others. The respondent interacts by using the most appropriate (synchronous or asynchronous) communication tools and their advanced functions (user groups, broadcast groups, mailing groups, distribution lists, web-based meetings, etc) effectively.

Creating and publishing digital content
Case 1: Question 7 The respondent selects and evaluates the most appropriate resources and applications for the objective pursued and the type of digital content to optimize creations.

Designing, integrating, and reworking digital content in various formats
Case 1: Question 3 The respondent creates and publishes complex content suited to the audiences, objectives, or purposes thereof, seeking the most appropriate content in each case (using a template).

Case 2: Question 5
The respondent uses and evaluates repositories of audiovisual content (images, audio recordings, GIFs a , videos, templates, etc) to design or rework digital content.

Protecting digital systems, devices, and content
Case 2: Question 9 The respondent protects digital files and content to prevent unauthorized access by third parties.

Protecting personal data and privacy
Case 2: Question 10 The respondent identifies critical points and suggests improvements for protecting personal data and privacy. Applying best practices, the respondent customizes the privacy settings of digital tools and environments and the permissions of apps to protect their identity and the privacy of the content they generate.

Acting in a civic manner in the digital environment
Case 2: Question 2 The respondent uses licenses and attribution systems suited to their objectives when publishing content in digital environments.
Case 2: Question 8 The respondent promotes coexistence in the digital environment.

Understanding the basics and using digital technology
Case 2: Question 11 The respondent uses digital technology and its environment autonomously.

Identifying personal and professional needs and applying digital solutions
Case 2: Question 6 The respondent is up to date with the latest technological trends and compares and evaluates digital devices and tools to select those that best meet their personal needs (leisure, health, protection, sports, emotional, etc) and professional needs (training, job search, productivity, and time management). a GIF: Graphics Interchange Format.

Data Collection
The survey was created and distributed to HPs using Microsoft Forms, together with an invitation email presenting the objective and characteristics of the study. Several meetings were held with HP associations and health service providers to explain our intention to conduct this study and that we would need their help to reach HPs. The invitation was distributed through the Catalan Department of Health and the Catalan Health Service, which sent it to the corresponding professional associations and to the human resources departments of public and private health care providers (hospitals, consortia, etc). Each institution decided how to disseminate the study among the professionals.
Before participating in the study, the participants had to provide consent for the study sponsor to process the information collected. The time taken to complete the survey (in minutes) was recorded.

Sample Size
The sample size was calculated so that it would be representative of the population of Catalan HPs (both in size and distribution). Of a total of 121,039 HPs [30], 36,520 (30.17%) were physicians, 45,995 (38%) were nurses, and 38,524 (31.83%) were other HPs. In 2019, 96,105/121,039 (79.40%) and 11,014/121,039 (9.10%) of these HPs worked in the Barcelona and Girona health care regions, respectively. The remaining (13,920/121,039, 11.50%) worked in other health care regions [14,30]. The minimum and maximum sample sizes were calculated using the Cochran formula, considering a 95% power and a 10% and 5% margin of error, respectively. This resulted in a sample size of 304 to 906 and 34 to 101 HPs in the Barcelona and Girona health care regions, respectively, and 45 to 134 HPs in other health care regions. The minimum and maximum overall sample sizes were in the range of 383 to 1141 HPs.

Statistical Analysis
Percentages and mean (SD) were used to summarize categorical and continuous variables, respectively. Categorical variables were compared using the chi-square test, whereas continuous variables were compared using the t test or ANOVA (for 2 or >2 comparators, respectively). These comparisons were 2-sided. The analysis of subgroups with a score of <25 in the digital competence test was performed using a 1-sided t test. Given that no maximum time to complete the survey was established and that the filling in of fields could be interrupted for personal or professional reasons, atypical observations where this was likely to have happened were removed to estimate a more realistic mean completion time. Outliers were filtered using the Hampel identifier. The internal consistency of the survey was analyzed using the greatest lower bound, given the lack of homogeneity of the scoring scale. The discriminatory index of each question of the ACTIC-derived digital competence test was calculated according to the study by Taib et al [18], where a score of ≥0.2 indicated good discrimination between HPs with intermediate and basic levels. Statistical analyses were performed using the R (version 4.11; R Foundation for Statistical Computing) software. A P value of <.05 was considered significant.

Ethics Approval
No ethics approval was required due to the type and nature of the study as the Catalan Department of Health is responsible for formulating the general criteria for health planning, setting the objectives, and the levels to be achieved in the topics that are included in the Health Plan for Catalonia [31]. All participants were informed about the study's purposes and that their participation was voluntary. Data protection treatment was informed to the participant and before accessing the survey, participants had to provide acceptance.

Results
Between June 28 and July 16, 2021, a total of 1009 potential participants accessed the survey, of whom 922 (91.40%) gave their consent to participating in it. Of these, 803 (79.6%) participants were classified as HPs according to the legal definition and constituted the study population. Table 2 shows the demographic and professional characteristics of the participants, most of whom were women (612/803, 76.2%), aged between 36 and 55 years (438/803, 54.5%). Nursing was the most common professional profile (227/803, 28.3%), followed by physicians (176/803, 21.9%). Nearly half of the participants worked in a subsidized private center. A total of (478/803, 59.5%) participants worked in specialized health care settings. Barcelona city was the most common work setting (209/803, 26%), followed by Camp de Tarragona (148/803, 18.4%). The mean length of professional experience was 19.6 (SD 11) years.  Disaggregated information relating to "Others" participants (biologists and dietitians or nutritionists, pharmacists, physiotherapists, dental hygienists, speech therapists, podiatrists, psychologists, and occupational therapists) can be found in Multimedia Appendix 2.

Digital Competence Test
The mean score in the digital competence test was 22.6 (SD 4.3), so most participants displayed a basic level (Table 4 and Figure 1). After removing outliers (122/803, 15%; see the Methods section), the mean time taken to complete the survey was 15.7 (SD 5.8) minutes.  The test score was higher in men (mean 23.2, SD 4.2, vs mean 22.4, SD 4.3 in women; P=.03), in younger participants (P<.001), and those with a high self-perceived level of digital competence (P<.001). We also observed differences among the various HP profiles, with other HP profiles scoring higher than nurses and physicians. Most of the analyzed subgroups had scores that were significantly <25 (Table 5).
A multivariate linear regression analysis confirmed the significant differences found in age, HP profile, and self-perception, but this was not confirmed for the gender variable.
In total, 7.1% (57/803) of participants had the ACTIC-2 certificate. Their mean score in the digital competence test was 23.4, SD 3.9), with 59.6% (34/57) having a basic level. The mean time taken to complete the survey was 11.8 (SD 3.6) minutes (Table 4). Of those 57 participants, 13 (22.8%) said their self-perceived knowledge was advanced, and 33 (57.9%) said their self-perceived knowledge was intermediate.
The internal consistency of the digital competence test as measured by greatest lower bound was 0.66 (acceptable consistency). The discrimination index of all questions except for one for all items was ≥0.2.

Principal Findings
Globally, there is a need to improve the digital skills of HPs through dedicated training to fully exploit the potential of digital technologies and to be able to provide the best possible care using such technologies [3,11,[32][33][34]. Multiple initiatives have been implemented to address this need [35]. For example, the EU*US eHealth Work Project devoted its efforts to identifying gaps and defining competences and developed a free introductory web-based course in eHealth [36]. Another major effort is that of Health Education England, which had defined a digital capability framework for improving the digital literacy of the health and care workforce and is currently testing a tool to self-assess digital literacy [37].
Our exploratory study in a large cohort representative of the Catalan HP population has provided valuable information regarding their digital competence level and training needs and revealed the consistency of the ad hoc digital competence test for the pursued objective. Nearly all participants (801/803, 99.8%) had either the basic or the intermediate level of competence, indicating that this HP group is already fulfilling the objectives set by the European Skills Agenda [38], whose aim is to ensure that at least 70% of adults have basic digital skills by 2025. However, the results should be interpreted with caution given the voluntary nature of participation, which might have given rise to a bias in assessing the participants' digital competence.
Some of the findings deserve further discussion. We only found statistically significant differences in the level of digital competence by age (higher in younger ages), self-rated digital competence (higher in those rated as advanced), and professional profile (higher in HP profiles other than nurses and physicians). Conversely, we found no differences based on workplace ownership (public vs private), level of health care, or work setting, which points to a homogeneous population in this sense. Most of the analyzed subgroups obtained an overall score that was statistically <25 (basic level). Exceptions to this included the youngest participants (aged 18-25 years); those with an advanced level of self-reported digital competence; and certain HP profiles such as biologists, dietitians or nutritionists, pharmacists, speech therapists, and clinical psychologists. Although it cannot be ruled out that these professional profiles may be more highly trained in digital competences as a result of their profession, the small size of these populations may have contributed to a biased result.
Our study also provides valuable information on the most frequently used professional digital tools (Office tools, social media, and electronic health records) and the most requested type of training (tools for disease prevention and health promotion, Office tools, electronic health records, and remote follow-up tools). The training needs were found to be broader than the nature of the main tools used, and these needs were inversely proportional to the frequency of use of specific digital tools. Both findings are indicative of HPs' interest in digital competence, which goes beyond the limitations imposed by their current professional digital skills or even by their current clinical practice. The results of an in-depth analysis of the most frequent HP profiles revealed interesting differences in the use of digital tools and training needs, with physicians displaying a broader range in terms of digital tools used and training needs [39].
Interestingly, only 7.1% (57/803) of the participants held an ACTIC-2 certificate, accrediting an intermediate level. This finding has several possible interpretations. On the one hand, the ACTIC certificate is voluntary and was conceived as a standard tool to prove citizens' digital competences when applying for a job or job promotion. This may explain its reduced representation in our sample and the fact that nearly 50% (394/803) of the HPs participating in our study were not aware of its existence. Not holding an ACTIC-2 certificate only denotes that the participants had not had access to this voluntary certificate. This may explain the similar scores observed between the participants with or without ACTIC-2. Of the 6.7% (54/803) of participants who rated their digital competence as advanced, only 24% (13/55) had the ACTIC-2 certificate. Objectively speaking, holding this certificate translated into a higher rate of participants scoring ≥25 (intermediate level: 23/57, 40%, vs 266/803, 33.1%) and lesser time taken to complete the survey (mean 11.8, SD 3.6 minutes vs mean 15.7, SD 5.8 minutes).
However, the statistical significance of these differences was not analyzed.
The findings of our study are highly valuable to the COMPDIG-Salut project as it establishes the basis for planning and deciding on specific strategic actions and policies to improve the digital competences of Catalan HPs. These are likely to be similar to other HPs in other EU countries. Furthermore, the results revealed the HPs' training needs from both professional and personal perspectives, which may serve as a starting point for designing tailored training actions.
Comparison with similar studies is hindered by differences in the populations included, the methodology, and the questionnaires used. Further analyses through an in-depth examination of the answers obtained for each of the formulated questions are currently underway to identify specific competence areas and on which training should be primarily focused.

Strengths and Limitations
Our study has several strengths, including a large sample size and sociodemographic representativeness of the main HP profiles in Catalonia. The distribution of the HPs' professional profiles was skewed with respect to the real Catalan scenario, with profiles of HPs other than nurses and physicians being overrepresented.
One of the novelties of our study is the way in which digital competence is assessed, that is, subjectively by self-efficacy (the most common method) and objectively by an ad hoc survey. Although the digital competency tool is yet to be fully validated, its acceptable consistency reinforces our findings and supports its validity for use in evaluating the digital competence level in the research setting. Owing to its approach, no barriers were anticipated for its local adaptation. Moreover, all the questions answered by the participants, except one, showed sufficient discriminatory power. However, given that the tool is a generic one aimed at all HPs, the adaptation of assessment scenarios and activities to different HP profiles will enable further refinement of the results. Resolving these weaknesses in the tool will improve its accuracy.
Our study also has several limitations, some of which have already been addressed. Assuming that the gold standard of testing should be a practical test and not a survey-based case test, some of these limitations are related to the study design: these mainly include aspects related to data collection, as no information was available for the percentage of people willing to participate, the differences between respondents and nonrespondents, including how they were approached, and the response rate. Other limitations are inherent to voluntary surveys, including concerns not only about the truthfulness of the answers provided and the attention placed on certain answers but also about selection bias, as we expect HPs with higher digital skills to have been more likely to participate in the study. Differences in the interpretation of questions posed should also be considered. Finally, the exploratory nature of this study leads to hypothesis-generating conclusions rather than definitive conclusions.

Conclusions
Knowing the digital competence level of HPs is fundamental for promoting relevant strategic policies and actions to ensure that the right resources and conditions are in place for good professional performance. Such strategies would include the design and provision of specific training to qualify and accredit the digital competences of HPs.
This exploratory study highlights the need to improve the digital competences of HPs working in Catalonia. The results have informed the Health Plan for Catalonia 2021-2025 [31] and lay the foundations for the development and deployment of a framework program for HPs' digital competences that should include assessment indicators and standards to meet the COMPDIG-Salut project's goals. On the basis of the definition of this digital competence framework, training methodologies and content will be developed for implementation in bachelor's degree programs (the basic educational level for students of the various health care professions) and continuing education programs for working HPs, which in both cases must include the assessment and accreditation of digital competences.
The digital competence test showed acceptable consistency for the objective pursued, although improvements are needed to fine-tune its accuracy. The findings of this study lay the foundations for designing a strategic plan for training Catalan HPs.