Our study was conducted in 3 distinct phases: (1) development of a tool to measure DVL, (2) collection of empiric cross-sectional data from a French adult population sample, and (3) assessment of the psychometric properties of the DVL tool.

We used the COSMIN (Consensus-Based Standards for the Selection of Health Measurement Instruments) to develop the DVL tool and validate it [23].

We based the conception of the DVL tool on the theories of digital health literacy and vaccine literacy, investigating the understanding, trust, appraisal, and application of vaccine-related information online [20,24], with the distinction between social media/forums and government websites. A panel of 5 public health researchers proposed a series of items inspired by the Health Literacy Questionnaire [25,26], the eHealth Literacy Scale [19], and the Vaccine Literacy Scale [22].

The construct of DVL was decided a priori and defined before any item activity. Expert judges confirmed through literature review that there were no existing instruments that will adequately serve the same purpose. A deductive method was used to identify the items through the description of the relevant field (domain), in combination with an inductive method based on the exchanges among experts. A group of 10 volunteers with characteristics similar to the target population pretested the questions. Items were worded in simple terms and unambiguously.

We narrowed the items focusing on vaccination and the digital environment to eventually obtain a total of 7 questions answered on a 4-point Likert scale (from 4 [agree] to 1 [disagree]) and an additional answer option “I do not know, I do not look for vaccine-related information.” This latter option was taken into account in the descriptions, but was considered “noninformative” for the analysis of the structural validity of the scale. The total score of the DVL scale was calculated through the sum of all answers to the items. The score of the scale varied from 7 to 28. The higher the score, the better the DVL level.

We also included an item on “the online sources which were the most consulted for vaccine-related information seeking” (online journals, government websites, health institution websites, social media, forums, video platforms, other). Finally, participants had to rate the importance of the use of the internet for vaccine-related information seeking through a visual analog scale from 1 (not important at all) to 5 (very important).

We administered the DVL tool to participants from an open online cohort (CONFINS) [27]. All participants were aged more than 18 years, living in France, and were able to read and understand French. CONFINS is a cohort collecting data on the impact of confinement on the health and well-being of the French population [28]. It included, among others, variables on opinions about vaccination and the DVL items. It also comprised sociodemographic information (age, gender, having children, being vaccinated against influenza) used in this study. Items were defined by a group of public health experts through several rounds of corrections and refinement. CONFINS consisted in a baseline questionnaire and repeated monthly follow-up questionnaires. Participants could decide whether to be contacted or not for the following phases of the survey. This study used data from the baseline questionnaire and the first follow-up questionnaire, covering the period from April to May 2020. This was a convenience sample.

CONFINS participants were recruited on a voluntary basis with no incentives through different communication channels. Posts were published on the social media (LinkedIn, Twitter, Facebook) of the University of Bordeaux and the partner contract research organization hosting the database. A total of 3 press releases were addressed to journalists. The coprinciple investigators were interviewed to promote the study. Three newsletters and weekly emails and SMS text messages were sent to the participants to remind them to complete the follow-up questionnaires. All recruitment strategies directed potential participants toward the CONFINS website including information on the objectives of the study and the investigators. Informed consent, containing details on the length of time of the survey, stored data, investigators and objectives of the study, was provided through an electronic signature.

Concerning the population of this study, we included all participants completing all items of the DVL tool, comprising also those choosing the answer option “I do not know, I do not look for vaccine-related information” (N=2935). However, for the sake of the specific analyses required to evaluate the psychometric properties of the DVL tool, we obtained a subsample of 848 participants who did not use the answer option “I do not know, I do not look for vaccine-related information.” The choice of using mainly the subsample was justified by the fact that the factor analysis mentioned later requires ordering the response modalities. As the “I do not know, I do not look for vaccine-related information” modality is difficult to classify, we decided to remove it. The subsample included those who had completed the baseline questionnaire (“test” phase). Among them, 62 participants also answered the follow-up questionnaire (“retest” phase).

First, a descriptive analysis of each item of the scale was performed for both the total sample of participants (N=2935) and the subsample (n=848). Participants of the subsample were also described according to their sociodemographic characteristics (ie, age, gender, working/studying in the field of health, having children, and being regularly vaccinated against flu). For quantitative variables, the mean and SD were calculated. For qualitative variables, participants were described in numbers and percentages. Answers to items were compared for each aforementioned sociodemographic characteristic. To do this, the item response options were grouped into “agree”/“rather agree” versus “disagree”/“rather disagree.” The statistical tests of χ² independence were used to compare the responses of the participants according to their sociodemographic criteria.

Second, an exploratory factor analysis (EFA) was performed on the baseline data to identify the underlying latent factors in the set of items as well as their association. As the items were ordinal variables, the polychoric correlation matrix of observed items was explored. Two initial hypotheses were tested. The first was the test of Bartlett sphericity. If the test was significant (P<.05), the observed matrix was significantly divergent from the null matrix and an EFA had to be performed. The second hypothesis required testing the measure of sampling adequacy using the Kaiser-Meyer-Olkin index [29]. This is a measure of the proportion of variance among the observed items, equivalent to the common variance. Thus, it was used to verify for partial correlations. If the Kaiser-Meyer-Olkin index was above 0.50, the EFA was adequate. Next, the number of factors to be kept in the model had to be chosen based on different criteria using eigenvalues. The Kaiser criterion consisted of keeping factors with eigenvalues greater than 1. The Cattell criterion (also called the “elbow criterion”) was based on identifying the inflection point, where the slope of the eigenvalue curve according to the number of factors in the model stabilized well below the “elbow.” Thus, the number of factors above the point was retained. The third criterion was the use of a parallel analysis. In this analysis, the eigenvalues obtained were compared with those that would be obtained from random data. The number of factors extracted was the number of factors whose eigenvalues were higher than those found with random data. In addition, the item × factor matrix had to be rotated to better identify how the items were substantially related to each factor. Among the several approaches to rotation, the oblique rotation was used because it considers the correlation between factors [30]. Finally, the items were associated with a factor when their saturation weight was close or superior to 0.30 and their communalities were considered as acceptable above 0.20. We also performed a confirmatory factor analysis (CFA) considering the criteria root-mean-square error of approximation (acceptable range between 0.08 and 0.1), comparative fit index (acceptable range >0.90) and standardized root-mean-square error (acceptable range between 0 and 0.008).

Third, to complete the validation of the DVL scale, the convergent and discriminant validities of the score were assessed. The sociodemographic criteria of participants with a low DVL score were compared with those of participants with a high score, determined according to the median, using χ² statistical tests of independence.

Statistical significance was considered if P<.05 and all tests were 2-tailed. Statistical analyses were performed on SAS version 9.3 software (SAS Institute).

The study was approved by the French Committee for the Protection of Individuals (Comité de Protection des Personnes [CPP], approval number 46-2020) and the French National Agency for Data Protection (Commission Nationale de l'Informatique et des Libertés [CNIL], approval number MLD/MFI/AR205600). The study follows the principles of the Declaration of Helsinki and the collection, storage, and analysis of the data comply with the European Union General Data Protection Regulation (EU GDPR).

Responses to the 7 items on the DVL tool by the total sample and the subsample are reported in Tables 1 and 2, respectively.

The “I do not know, I do not look for vaccine-related information” response rates were 51.99% (1526/2935) for item 1, 22.76% (668/2935) for item 2, 1.36% (40/2935) for item 3, 16.73% (491/2935) for item 4, 38.13% (1119/2935) for item 5, 5.04% (148/2935) for item 6, and 24.67% (724/2935) for item 7. Per participant, the maximum number of “I do not know, I do not look for vaccine-related information” was 5; 24.74% (726/2935) responded “I do not know, I do not look for vaccine-related information” for at least one item; 23.51% (690/2395) for at least two items; 10.97% (322/2935) for at least three items; 7.97% (234/2935) for at least four items; and 3.92% (115/2395) for at least five items. The mean of responses per participant was 1.56 (SD 1.4). In addition, the use of a factor analysis requires ordering the response modalities. As the “I do not know, I do not look for vaccine-related information” modality is difficult to classify in view of the others, we decided to remove it from the analyses. Therefore, the study sample contained 848 participants who responded to the items as shown in Table 2.

All item response options were used, thus qualifying them as informative. In addition, Table 2 shows that the items were discriminating because the response rates for each modality were in the average. The intraclass correlation coefficient (ICC) was calculated based on data from the 62 participants. Items 1, 5, and 7 presented a low ICC, which could be explained by nonconcordant responses between the 2 measurements, and therefore less reliability, their formulation, and possible difficulty in answering them. In fact, these items had the highest percentages of the “I do not know, I do not look for vaccine-related information” responses (Table 1).

In the subsample of 848 participants, 73.1% (620/848) were females. The mean age was 29.9 (SD 12.3). Participants working or studying in the field of health were 397/848 (46.8%). The percentage of parents was 20.9% (178/848) and 557/848 (65.7%) were not vaccinated against flu (Table 3).

The mean of the importance of the use of the internet for vaccine-related information seeking was 3.7 out of 5 (SD 1.1). The most used source for vaccine-related information seeking was websites of health institutions (395/848, 46.6%), followed by government websites (184/848, 21.7%). Online journals were consulted by 56/848 individuals (6.6%), whereas other sources by 37/848 individuals (4.4%). Social networks were consulted by 70/848 individuals (8.3%), video platforms by 16/848 (1.9%), and forums by 8/848 (0.9%).

Multimedia Appendix 2 reports data on the comparison of the answer to the DVL items according to sociodemographic characteristics.

Regarding their answers to the items, women were more in agreement with the statement of item 3 (I can detect vaccine-related fake news), item 4 (I trust vaccine-related information provided by government websites), and item 7 (I think the information I find online may influence my decision to get vaccinated) than men. Participants aged 35 or over disagreed with item 1 (I find vaccine-related information on social media and forums is understandable), which was different from those under 35 years. Participants studying or working in the field of health and those receiving regular flu shots were more in agreement with items 2 (I find vaccine-related information on government websites is understandable), item 3 (I can detect vaccine-related fake news), and item 4 (I trust vaccine-related information provided by government websites) and disagreed with item 7 (I think the information I find online may influence my decision to get vaccinated) compared with those who worked or studied in another field and those who did not get a flu shot. There was no difference in responses concerning parenthood.

The interitem polychoric correlation matrix was used for the first definition of the associations between items (Table 4).

In the polychoric matrix, we observed strong correlations between items 2, 3, and 4. Item 1 was more correlated with item 5.

The hypotheses justifying the performance of an EFA were validated. The Bartlett test of sphericity showed a P<.05 (χ²₂₁=1319.37) and the Kaiser-Meyer-Olkin index was 0.58, indicating good sampling adequacy.

The number of factors was calculated based on the Kaiser and Cattell criteria and the parallel analysis; 3 factors were kept (Figure 1).

Finally, several EFAs were performed to test the different oblique rotations. The OBLIMIN oblique rotation was the most common. Table 5 shows that items 1 and 5 were associated with factor 2; items 2, 3, and 4 with factor 1; and items 6 and 7 with factor 3. The oblique rotation OBEAQUAMAX showed that saturation weights revealed several possible associations between items and factors. Items 3 and 7 were associated with both factors 1 and 3 based on the saturation weights close or superior to 0.30. Communalities were all acceptable.

Table 6 shows the interfactor correlations according to the OBLIMIN and OBEAQUAMAX rotations. Correlations were low but factor 1 was negatively correlated with factor 2, and factor 3 was positively correlated with the other 2 factors.

In view of these results, the relationships between the items and the factors were interpreted as follows. Factor 1 was associated with items relating to “reliable” information about vaccination (government sites), with the label “understanding and trust official information about vaccination provided by institutional websites.” Factor 2 was associated with items related to information about vaccination of which 1 should be relatively “unreliable” (social media) with the label “understanding and trust information about vaccines as provided by social media.” Finally, factor 3 was associated with items related to the application of knowledge on vaccination consulted on the web (label of factor 3).

Finally, we also performed a CFA to confirm these 3 dimensions (Table 7).

In the CFA the criterion values were as follows: root-mean-square error of approximation 0.12 (90% CI 0.11-1.14), comparative fit index 0.80, and standardized root-mean-square error 0.08.

The mean DVL score of the baseline sample of 848 participants was 19.5 (SD 2.8). Participants scored between 14 and 21 points (ie, in the medium DVL range). The median was 20.

Table 8 shows the sociodemographic characteristics of the sample according to the DVL level. The score was dichotomized into <20 (low DVL score) and ≥20 (high DVL score).

Participants with a low DVL level were significantly older (30.8 years vs 29 years; P=.03). Those working or studying in the field of health were significantly more numerous in the group with a higher score (P=.01). Those who did not receive regular flu vaccinations were significantly more likely to be in the low score group (P=.01). Among online sources for vaccine-related information, government websites were more used by those with a higher DVL (P=.03). Those with a score less than 20 considered the use of the internet for vaccine-related information less important than others, with the means being 3.4 (SD 1.1) and 4.0 (0.9), respectively.

We conceived a scale measuring DVL and assessed its psychometric proprieties among a sample of French adults. The scale was composed of 7 items covering the overarching construct of DVL, which includes 3 subdimensions. The first subdimension (items 2 and 4) refers to understanding and trusting official information about vaccination provided by institutional websites. The second subdimension (items 1 and 5) refers to understanding and trusting information about vaccines as provided by social media. The underlying assumption for these 2 dimensions is that government websites provide valid information while social media provide fake news [31]. In this line, in our sample, the most accessed sources were health institutions and government websites, while social media and forums were less consulted.

The third subdimension (items 3, 6, and 7) refers to the appraisal of vaccine information online in terms of evaluation of the information and its application for decision making. Two items (3 and 7) are actually included in both subdimensions 1 and 2. For the item “I can detect fake news,” this ambivalence can be explained by the fact that recognizing fake news is a reflection of both the understanding/trust of official information (subdimension 1) and the appraisal and practical application of found information (subdimension 3). The possible explanation is that those who recognize fake news are more inclined to government websites and are more cautious in interpreting vaccine-related information. The inclusion of the item “I think the information I find online may influence my decision to get vaccinated” in both subdimensions 1 and 3 can be interpreted as the fact that trusting official information might correspond to a higher capacity to make correct evidence-based decisions about vaccination. This overlap of factors infers an interrelation of items, which can suggest that the scale is coherent and congruent.

Some recommendations must be considered when using the DVL scale. There are 4 response options (disagree, rather disagree, rather agree, and agree) that are used to obtain a score. However, even if it does not contribute to the calculation of the score, the fifth response option (I do not know, I do not look for vaccine-related information) provides useful information. First, this option respects the opinion of those not feeling concerned without forcing or biasing their answer. Second, it is really interesting to measure the percentage of those who do not feel concerned by seeking vaccine-related information online. In this study, one-half of the participants used the option “I do not know, I do not look for vaccine-related information” for the item on understanding information found on social media, and more than one-third for the item on trust in social media. These results confirm the fact that social media are more rarely used than government websites for this type of information. Thus, we suggest to calculate the score by considering as missing values all cases including 1 response option “I do not know, I do not look for vaccine-related information”, and to complete this information with the percentage of those using this same option. These data are complementary in measuring DVL.

Having a low DVL score (<20) can be interpreted as a relevant alarm in relation to the extensive use of the internet for vaccine-related contents, especially in France [15]. As is the case with health literacy, low DVL scores are associated with a higher risk of adopting an unhealthy behavior [32]; in this case this refers to the decision of not to get vaccinated. Not being able to navigate information on the internet could increase the chance of having a negative perception about vaccines [33]. Lower scores in the scale would also correspond to the incapacity to recognize fake news and trust in unofficial information provided by social media. There are many who consult the internet regarding vaccination and it is important to know their levels of DVL to help them navigate online information.

DVL scores were significantly different by age (participants with a low DVL score were significantly older), studying or working in the field of health (those working or studying in the field of health were significantly more numerous in the group with a high score), and being vaccinated against flu (those who did not regularly get vaccinated against influenza were significantly more numerous in the group with a low score). These results are in line with previous literature concerning general health literacy: scores of health literacy are higher in younger adults [34], health care professionals [35], and those vaccinated against flu [36].

Comparison with results from other studies is not possible because DVL has never been measured before.

This study is the very first to develop and validate a standardized instrument for assessing general DVL in people. It responds to the urgent need for similar scales to tackle vaccine-related misinformation [37], especially in relation to the COVID-19 pandemic. Measuring the DVL of individuals consulting the internet for information on COVID-19–related vaccination could inform health institutions, communication experts, and health care providers to plan and implement strategies to overcome gaps in DVL and promote vaccination [38]. Furthermore, analyses performed in this study are robust and based on an in-depth knowledge of psychometrics techniques. In particular, the use of the bifactorial model is justified by the fact that it considers correlations between items based on the general factor and the relations between the general factor. Items are not limited by the group factors. This model is largely applied in cognitive and psychological sciences [39].

This study is not without limitations. Items were defined a priori based on existing scales but limited to 7. A larger number of items might have provided a more exhaustive coverage of DVL factors. The population under study was not representative of French adults given that it comprised a high number of women (2971/3738, 79.48%), students (3498/3783, 93.58%), and young people (29.2 years) [40], compared with the general population [41]. However, the sample was large enough to assess the relevance of the scale. Low ICC values in some separated items might be explained by an inaccurate phrasing. The ICCs of 3 items were low, which corresponds to a low reliability. Future instruments might be based on our scale, but we propose more precise wording according to the population of interest in a specific context (eg, cultural or sociodemographic characteristics).

The DVL scale is the first instrument providing information on the way individuals understand, trust, and appraise vaccine-related information on the internet through 2 channels, namely, social media and government websites. The DVL scale has good psychometric properties in terms of content validity, dimensionality, and convergent and discriminant validity. Results show that the scale can be easily administered with well-grounded outcomes. It is a screening instrument contributing to detect people who need to be supported in navigating vaccine-related information online. It can be used in questionnaires to identify profiles of web users who could be influenced by anti-vax movements, for instance. Providing the instructions to look for online information and to understand its content is the key to spreading good vaccine-related information and promoting vaccination in general [42]. The scale can be used to measure DVL in the French population and translated validated versions could be proposed internationally.