This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.
The use of the internet to look for information about vaccines has skyrocketed in the last years, especially with the COVID-19 pandemic. Digital vaccine literacy (DVL) refers to understanding, trust, appraisal, and application of vaccine-related information online.
This study aims to develop a tool measuring DVL and assess its psychometric properties.
A 7-item online questionnaire was administered to 848 French adults. Different psychometric analyses were performed, including descriptive statistics, exploratory factor analysis, confirmatory factor analysis, and convergent and discriminant validity.
We developed the 7-item DVL scale composed of 3 factors (understanding and trust official information; understanding and trust information in social media; and appraisal of vaccine information online in terms of evaluation of the information and its application for decision making). The mean DVL score of the baseline sample of 848 participants was 19.5 (SD 2.8) with a range of 7-28. The median score was 20. Scores were significantly different by gender (
The DVL tool showed good psychometric proprieties, resulting in a promising measure of DVL.
Vaccination is one of the most commonly queried topics on the internet [
Online sources for vaccine-related information vary. These include websites of official institutions, blogs, forums, social media, among others. The information they convey can be either reliable and valid or unscientific and misleading. On the one hand, social media have been defined as a powerful catalyst for the “anti-vax movement” [
Hesitancy toward vaccination remains a present and growing issue [
Preliminary studies have explored the influence of the internet on growing vaccine hesitancy [
Digital health literacy refers to the capacity of people to adequately understand and process online health information to meet their needs [
A valid tool for measurement of DVL is thus essential to provide inputs to train people in better navigating vaccine-related information on the internet on both social media and official online sources. This scale developed herein also allows to provide a general and population-based assessment of DVL: given the spread of the COVID-19 pandemic and the relevance of accepting vaccination, today more than ever it is pivotal to investigate the level of DVL in the population and examine its potential contribution to vaccine uptake. Furthermore, the scale can be used as an instrument to measure the effectiveness of interventions aimed at increasing DVL for reducing vaccine hesitancy.
To the best of our knowledge, no tool exists to measure DVL. The currently used questionnaires focus on vaccine literacy in general and not on online vaccine literacy (ie, DVL) [
Our study was conducted in 3 distinct phases: (1) development of a tool to measure DVL, (2) collection of empiric cross-sectional data from a French adult population sample, and (3) assessment of the psychometric properties of the DVL tool.
We used the COSMIN (Consensus-Based Standards for the Selection of Health Measurement Instruments) to develop the DVL tool and validate it [
We based the conception of the DVL tool on the theories of digital health literacy and vaccine literacy, investigating the understanding, trust, appraisal, and application of vaccine-related information online [
The construct of DVL was decided a priori and defined before any item activity. Expert judges confirmed through literature review that there were no existing instruments that will adequately serve the same purpose. A deductive method was used to identify the items through the description of the relevant field (domain), in combination with an inductive method based on the exchanges among experts. A group of 10 volunteers with characteristics similar to the target population pretested the questions. Items were worded in simple terms and unambiguously.
We narrowed the items focusing on vaccination and the digital environment to eventually obtain a total of 7 questions answered on a 4-point Likert scale (from 4 [agree] to 1 [disagree]) and an additional answer option “I do not know, I do not look for vaccine-related information.” This latter option was taken into account in the descriptions, but was considered “noninformative” for the analysis of the structural validity of the scale. The total score of the DVL scale was calculated through the sum of all answers to the items. The score of the scale varied from 7 to 28. The higher the score, the better the DVL level.
We also included an item on “the online sources which were the most consulted for vaccine-related information seeking” (online journals, government websites, health institution websites, social media, forums, video platforms, other). Finally, participants had to rate the importance of the use of the internet for vaccine-related information seeking through a visual analog scale from 1 (not important at all) to 5 (very important).
We administered the DVL tool to participants from an open online cohort (CONFINS) [
CONFINS participants were recruited on a voluntary basis with no incentives through different communication channels. Posts were published on the social media (LinkedIn, Twitter, Facebook) of the University of Bordeaux and the partner contract research organization hosting the database. A total of 3 press releases were addressed to journalists. The coprinciple investigators were interviewed to promote the study. Three newsletters and weekly emails and SMS text messages were sent to the participants to remind them to complete the follow-up questionnaires. All recruitment strategies directed potential participants toward the CONFINS website including information on the objectives of the study and the investigators. Informed consent, containing details on the length of time of the survey, stored data, investigators and objectives of the study, was provided through an electronic signature.
Concerning the population of this study, we included all participants completing all items of the DVL tool, comprising also those choosing the answer option “I do not know, I do not look for vaccine-related information” (N=2935). However, for the sake of the specific analyses required to evaluate the psychometric properties of the DVL tool, we obtained a subsample of 848 participants who did not use the answer option “I do not know, I do not look for vaccine-related information.” The choice of using mainly the subsample was justified by the fact that the factor analysis mentioned later requires ordering the response modalities. As the “I do not know, I do not look for vaccine-related information” modality is difficult to classify, we decided to remove it. The subsample included those who had completed the baseline questionnaire (“test” phase). Among them, 62 participants also answered the follow-up questionnaire (“retest” phase).
First, a descriptive analysis of each item of the scale was performed for both the total sample of participants (N=2935) and the subsample (n=848). Participants of the subsample were also described according to their sociodemographic characteristics (ie, age, gender, working/studying in the field of health, having children, and being regularly vaccinated against flu). For quantitative variables, the mean and SD were calculated. For qualitative variables, participants were described in numbers and percentages. Answers to items were compared for each aforementioned sociodemographic characteristic. To do this, the item response options were grouped into “agree”/“rather agree” versus “disagree”/“rather disagree.” The statistical tests of
Second, an exploratory factor analysis (EFA) was performed on the baseline data to identify the underlying latent factors in the set of items as well as their association. As the items were ordinal variables, the polychoric correlation matrix of observed items was explored. Two initial hypotheses were tested. The first was the test of Bartlett sphericity. If the test was significant (
Third, to complete the validation of the DVL scale, the convergent and discriminant validities of the score were assessed. The sociodemographic criteria of participants with a low DVL score were compared with those of participants with a high score, determined according to the median, using
Statistical significance was considered if
The study was approved by the French Committee for the Protection of Individuals (Comité de Protection des Personnes [CPP], approval number 46-2020) and the French National Agency for Data Protection (Commission Nationale de l'Informatique et des Libertés [CNIL], approval number MLD/MFI/AR205600). The study follows the principles of the Declaration of Helsinki and the collection, storage, and analysis of the data comply with the European Union General Data Protection Regulation (EU GDPR).
Responses to the 7 items on the DVL tool by the total sample and the subsample are reported in
Results of all potentials items of the DVL scalea in the CONFINS online cohort (N=2935).
Items | Disagree, n (%) | Rather disagree, n (%) | Rather agree, n (%) | Agree, n (%) | Do not know, n (%) |
1. I find vaccine-related information on social media and forums is understandable | 215 (7.33) | 478 (16.29) | 582 (19.83) | 134 (4.57) | 1526 (51.99) |
2. I find vaccine-related information on government websites is understandable | 111 (3.78) | 176 (6) | 1394 (47.50) | 586 (19.97) | 668 (22.76) |
3. I can detect vaccine-related fake news | 97 (3.30) | 477 (16.25) | 1500 (51.11) | 821 (27.97) | 40 (1.36) |
4. I trust vaccine-related information provided by government websites | 55 (1.87) | 191 (6.51) | 1250 (42.59) | 948 (32.30) | 491 (16.73) |
5. I find vaccine-related information on social networks is valid | 533 (18.16) | 1123 (38.26) | 134 (4.53) | 26 (0.89) | 1119 (38.13) |
6. When I read vaccination information online, I cross-reference it with other sources to verify its validity | 178 (6.06) | 394 (13.42) | 1288 (43.88) | 1060 (36.12) | 15 (0.51) |
7. I think the information I find online may influence my decision to get vaccinated | 413 (14.07) | 649 (22.11) | 918 (31.28) | 231 (7.97) | 724 (24.67) |
aDVL scale: Digital Vaccine Literacy scale.
Results of all potential items of the DVL scalea in the CONFINS online cohort (n=848, without “do not know”).
Item | Disagree, n (%) | Rather disagree, n (%) | Rather agree, n (%) | Agree, n (%) | Test-retest reliability (n=62), intraclass correlation coefficient (95% CI) |
1. I find vaccine-related information on social media and forums is understandable | 139 (16.4) | 287 (33.8) | 342 (40.3) | 80 (9.4) | 0.14 (0.01 to 0.37) |
2. I find vaccine-related information on government websites is understandable | 49 (5.8) | 82 (9.7) | 492 (58.0) | 225 (26.5) | 0.53 (0.33 to 0.69) |
3. I can detect vaccine-related |
27 (3.2) | 111 (13.1) | 421 (49.6) | 289 (34.1) | 0.70 (0.55 to 0.81) |
4. I trust vaccine-related information provided by government websites | 23 (2.7) | 82 (9.7) | 409 (48.2) | 334 (39.4) | 0.46 (0.24 to 0.63) |
5. I find vaccine-related information on social networks is valid | 224 (26.4) | 529 (62.4) | 83 (9.8) | 12 (1.4) | 0.05 (0.01 to 0.29) |
6. When I read vaccination information online, I cross-reference it with other sources to verify its validity | 44 (5.2) | 87 (10.3) | 365 (43) | 352 (41.5) | 0.48 (0.27 to 0.65) |
7. I think the information I find online may influence my decision to get vaccinated | 122 (14.4) | 267 (31.5) | 354 (41.7) | 105 (12.4) | –0.09 (–0.33 to 0.16) |
aDVL scale: Digital Vaccine Literacy scale.
The “I do not know, I do not look for vaccine-related information” response rates were 51.99% (1526/2935) for item 1, 22.76% (668/2935) for item 2, 1.36% (40/2935) for item 3, 16.73% (491/2935) for item 4, 38.13% (1119/2935) for item 5, 5.04% (148/2935) for item 6, and 24.67% (724/2935) for item 7. Per participant, the maximum number of “I do not know, I do not look for vaccine-related information” was 5; 24.74% (726/2935) responded “I do not know, I do not look for vaccine-related information” for at least one item; 23.51% (690/2395) for at least two items; 10.97% (322/2935) for at least three items; 7.97% (234/2935) for at least four items; and 3.92% (115/2395) for at least five items. The mean of responses per participant was 1.56 (SD 1.4). In addition, the use of a factor analysis requires ordering the response modalities. As the “I do not know, I do not look for vaccine-related information” modality is difficult to classify in view of the others, we decided to remove it from the analyses. Therefore, the study sample contained 848 participants who responded to the items as shown in
All item response options were used, thus qualifying them as informative. In addition,
In the subsample of 848 participants, 73.1% (620/848) were females. The mean age was 29.9 (SD 12.3). Participants working or studying in the field of health were 397/848 (46.8%). The percentage of parents was 20.9% (178/848) and 557/848 (65.7%) were not vaccinated against flu (
The mean of the importance of the use of the internet for vaccine-related information seeking was 3.7 out of 5 (SD 1.1). The most used source for vaccine-related information seeking was websites of health institutions (395/848, 46.6%), followed by government websites (184/848, 21.7%). Online journals were consulted by 56/848 individuals (6.6%), whereas other sources by 37/848 individuals (4.4%). Social networks were consulted by 70/848 individuals (8.3%), video platforms by 16/848 (1.9%), and forums by 8/848 (0.9%).
Regarding their answers to the items, women were more in agreement with the statement of item 3 (I can detect vaccine-related fake news), item 4 (I trust vaccine-related information provided by government websites), and item 7 (I think the information I find online may influence my decision to get vaccinated) than men. Participants aged 35 or over disagreed with item 1 (I find vaccine-related information on social media and forums is understandable), which was different from those under 35 years. Participants studying or working in the field of health and those receiving regular flu shots were more in agreement with items 2 (I find vaccine-related information on government websites is understandable), item 3 (I can detect vaccine-related fake news), and item 4 (I trust vaccine-related information provided by government websites) and disagreed with item 7 (I think the information I find online may influence my decision to get vaccinated) compared with those who worked or studied in another field and those who did not get a flu shot. There was no difference in responses concerning parenthood.
Sociodemographic characteristics of the CONFINS study population.
Characteristics | Value | |
Age, mean (SD) | 29.9 (12.3) | |
|
|
|
|
18-34 | 653 (78.2) |
|
≥35 | 182 (21.8) |
|
|
|
|
Female | 620 (73.1) |
|
Male | 228 (26.9) |
|
|
|
|
No | 366 (48.0) |
|
Yes | 397 (52.0) |
|
|
|
|
No | 670 (79.0) |
|
Yes | 178 (21.0) |
|
|
|
|
No | 557 (65.7) |
|
Yes | 291 (34.3) |
The interitem polychoric correlation matrix was used for the first definition of the associations between items (
In the polychoric matrix, we observed strong correlations between items 2, 3, and 4. Item 1 was more correlated with item 5.
The hypotheses justifying the performance of an EFA were validated. The Bartlett test of sphericity showed a
The number of factors was calculated based on the Kaiser and Cattell criteria and the parallel analysis; 3 factors were kept (
Finally, several EFAs were performed to test the different oblique rotations. The OBLIMIN oblique rotation was the most common.
Interitem polychoric correlation matrix.
Item | 1 | 2 | 3 | 4 | 5 | 6 | 7 |
1 | —a | — | — | — | — | — | — |
2 | 0.33 | — | — | — | — | — | — |
3 | 0.00 | 0.46 | — | — | — | — | — |
4 | 0.06 | 0.64 | 0.52 | — | — | — | — |
5 | 0.45 | –0.02 | –0.10 | –0.06 | — | — | — |
6 | 0.06 | 0.19 | 0.34 | 0.12 | –0.02 | — | — |
7 | 0.13 | –0.11 | –0.13 | –0.15 | 0.21 | 0.20 | — |
aDashes correspond to the absence of a correlation between items.
Distribution of the median simulated eigenvalues according to the number of factors and application of the parallel analysis. 7 variables, iterations, 848 observations.
Matrices of the saturation weights with oblique rotations and item communalities.
Item | OBLIMIN | OBEAQUAMAX | Communality | ||||
|
Factor 1 | Factor 2 | Factor 3 | Factor 1 | Factor 2 | Factor 3 | |
1 | 0.19 | 0.69 | –0.02 | 0.19 | 0.67 | 0.01 | 0.46 |
2 | 0.78 | 0.23 | –0.01 | 0.74 | 0.21 | 0.13 | 0.63 |
3 | 0.60 | –0.14 | 0.25 | 0.50 | –0.15 | 0.37 | 0.47 |
4 | 0.76 | 0.01 | –0.03 | 0.72 | –0.01 | 0.12 | 0.57 |
5 | –0.08 | 0.56 | 0.03 | –0.07 | 0.57 | –0.01 | 0.34 |
6 | 0.17 | –0.05 | 0.49 | 0.03 | –0.04 | 0.53 | 0.28 |
7 | –0.23 | 0.20 | 0.33 | -0.30 | 0.21 | 0.29 | 0.21 |
In view of these results, the relationships between the items and the factors were interpreted as follows. Factor 1 was associated with items relating to “reliable” information about vaccination (government sites), with the label “understanding and trust official information about vaccination provided by institutional websites.” Factor 2 was associated with items related to information about vaccination of which 1 should be relatively “unreliable” (social media) with the label “understanding and trust information about vaccines as provided by social media.” Finally, factor 3 was associated with items related to the application of knowledge on vaccination consulted on the web (label of factor 3).
Finally, we also performed a CFA to confirm these 3 dimensions (
In the CFA the criterion values were as follows: root-mean-square error of approximation 0.12 (90% CI 0.11-1.14), comparative fit index 0.80, and standardized root-mean-square error 0.08.
Interfactor correlation matrices (OBLIMIN and OBEAQUAMAX).
Factor | OBLIMIN | OBEAQUAMAX | ||||
Factor 1 | Factor 2 | Factor 3 | Factor 1 | Factor 2 | Factor 3 | |
1 | 1 | —a | — | 1 | — | — |
2 | –0.08 | 1 | — | –0.09 | 1 | — |
3 | 0.11 | 0.18 | 1 | 0.19 | 0.16 | 1 |
aDashes correspond to the absence of a correlation between items and factors.
Weights of the relationships item-factors of the final model by confirmatory factor analysis.
Item | Model 1 | ||
|
Factor 1 | Factor 2 | Factor 3 |
1 | —a | 0.87 | — |
2 | 0.56 | — | — |
3 | 0.43 | — | — |
4 | 0.51 | — | — |
5 | — | 0.23 | — |
6 | — | — | 0.83 |
7 | — | — | 0.15 |
aDashes correspond to the absence of a correlation between items and factors.
The mean DVL score of the baseline sample of 848 participants was 19.5 (SD 2.8). Participants scored between 14 and 21 points (ie, in the medium DVL range). The median was 20.
Participants with a low DVL level were significantly older (30.8 years vs 29 years;
Sociodemographic characteristics of the baseline sample by DVLa level (n=848).b
Sociodemographics | Low DVL (score <20) | High DVL (score ≥20) | |||||
Age (years), mean (SD) | 30.8 (12.9) | 29.0 (11.7) | .03 | ||||
|
|
|
.04 | ||||
|
18-34 | 298/397 (75.1) | 355/438 (81.1) |
|
|||
|
≥35 | 99/397 (24.9) | 83/438 (18.9) |
|
|||
|
|
|
.24 | ||||
|
Female | 303/404 (75) | 317/444 (71.4) |
|
|||
|
Male | 101/404 (25) | 127/444 (28.6) |
|
|||
|
|
|
.01 | ||||
|
No | 192/357 (53.8) | 174/406 (42.9) |
|
|||
|
Yes | 165/357 (46.2) | 232/406 (57.1) |
|
|||
|
|
|
.38 | ||||
|
No | 314/404 (77.7) | 356/444 (80.2) |
|
|||
|
Yes | 90/404 (22.3) | 88/444 (19.8) |
|
|||
|
|
|
.01 | ||||
|
No | 283/404 (70) | 274/444 (61.7) |
|
|||
|
Yes | 121/404 (30) | 170/444 (38.3) |
|
|||
|
|
|
.03 | ||||
|
Online journals | 30/338 (8.9) | 26/390 (6.7) |
|
|||
|
Government websites | 73/338 (21.6) | 111/390 (28.5) |
|
|||
|
Health institutions websites | 185/338 (54.7) | 210/390 (53.8) |
|
|||
|
Social media | 19/338 (5.6) | 13/390 (3.3) |
|
|||
|
Forums | 7/338 (2.1) | 1/390 (0.3) |
|
|||
|
Video Platforms | 5/338 (1.5) | 11/390 (2.8) |
|
|||
|
Other | 19/338 (5.6) | 18/390 (4.6) |
|
|||
Importance of the use of the internet for vaccine-related information seeking (n=338), mean (SD) | 3.4 (1.1)c | 4.0 (0.9)d | <.001 |
aDVL: digital vaccine literacy.
bValues are presented as n/N (%) unless indicated otherwise.
cN=338.
dN=390.
We conceived a scale measuring DVL and assessed its psychometric proprieties among a sample of French adults. The scale was composed of 7 items covering the overarching construct of DVL, which includes 3 subdimensions. The first subdimension (items 2 and 4) refers to understanding and trusting official information about vaccination provided by institutional websites. The second subdimension (items 1 and 5) refers to understanding and trusting information about vaccines as provided by social media. The underlying assumption for these 2 dimensions is that government websites provide valid information while social media provide fake news [
The third subdimension (items 3, 6, and 7) refers to the appraisal of vaccine information online in terms of evaluation of the information and its application for decision making. Two items (3 and 7) are actually included in both subdimensions 1 and 2. For the item “I can detect fake news,” this ambivalence can be explained by the fact that recognizing fake news is a reflection of both the understanding/trust of official information (subdimension 1) and the appraisal and practical application of found information (subdimension 3). The possible explanation is that those who recognize fake news are more inclined to government websites and are more cautious in interpreting vaccine-related information. The inclusion of the item “I think the information I find online may influence my decision to get vaccinated” in both subdimensions 1 and 3 can be interpreted as the fact that trusting official information might correspond to a higher capacity to make correct evidence-based decisions about vaccination. This overlap of factors infers an interrelation of items, which can suggest that the scale is coherent and congruent.
Some recommendations must be considered when using the DVL scale. There are 4 response options (
Having a low DVL score (<20) can be interpreted as a relevant alarm in relation to the extensive use of the internet for vaccine-related contents, especially in France [
DVL scores were significantly different by age (participants with a low DVL score were significantly older), studying or working in the field of health (those working or studying in the field of health were significantly more numerous in the group with a high score), and being vaccinated against flu (those who did not regularly get vaccinated against influenza were significantly more numerous in the group with a low score). These results are in line with previous literature concerning general health literacy: scores of health literacy are higher in younger adults [
Comparison with results from other studies is not possible because DVL has never been measured before.
This study is the very first to develop and validate a standardized instrument for assessing general DVL in people. It responds to the urgent need for similar scales to tackle vaccine-related misinformation [
This study is not without limitations. Items were defined a priori based on existing scales but limited to 7. A larger number of items might have provided a more exhaustive coverage of DVL factors. The population under study was not representative of French adults given that it comprised a high number of women (2971/3738, 79.48%), students (3498/3783, 93.58%), and young people (29.2 years) [
The DVL scale is the first instrument providing information on the way individuals understand, trust, and appraise vaccine-related information on the internet through 2 channels, namely, social media and government websites. The DVL scale has good psychometric properties in terms of content validity, dimensionality, and convergent and discriminant validity. Results show that the scale can be easily administered with well-grounded outcomes. It is a screening instrument contributing to detect people who need to be supported in navigating vaccine-related information online. It can be used in questionnaires to identify profiles of web users who could be influenced by anti-vax movements, for instance. Providing the instructions to look for online information and to understand its content is the key to spreading good vaccine-related information and promoting vaccination in general [
Original items of the DVL scale (French). DVL scale: Digital Vaccine Literacy scale.
Comparison of responses to the 7 DVL items according to sociodemographic characteristics (n=848). DVL: digital vaccine literacy.
confirmatory factor analysis
Commission Nationale de l'Informatique et des Libertés
Consensus-Based Standards for the Selection of Health Measurement Instruments
Comité de Protection des Personnes
digital vaccine literacy
exploratory factor analysis
European Union General Data Protection Regulation
intraclass correlation coefficient
We wish to thank all members of the CONFINS group including the i-Share, Kappa Santé, and Kap Code team members: we especially acknowledge Garance Perret and Mathilde Pouriel for data analysis; Julie Arsandaux, Shérazade Kinouani, and Mélissa Macalli for paper writing; Raphaël Germain and Clothilde Pollet for regulatory affairs; and Vanessa Marie-Joseph, Adel Mebarki, Elena Milesi, and Marie Mougin for the study communication. Kevin Ouazzani Touhami is also gratefully acknowledged. The authors are also grateful to all the participants who volunteered to take part in the study. The i-Share team is currently supported by an unrestricted grant of the Nouvelle-Aquitaine Regional Council (Conseil Régional Nouvelle-Aquitaine, grant N°4370420). It has also received grants from the Nouvelle-Aquitaine Regional Health Agency (Agence Régionale de Santé Nouvelle-Aquitaine, grant N°6066R-8), Public Health France (Santé Publique France, grant N°19DPPP023-0), and The National Institute against cancer INCa (grant N°INCa_11502). The article fees were covered by the Plan Propio - UCA 2022-2023, and the RÉFLIS network. The funding bodies were neither involved in the study design, or in the collection, analysis, or interpretation of the data.
All data generated or analyzed during this study are included in this published article. The full data set is available upon request from the CONFINS cohort team.
IM conceived the study and wrote and revised the manuscript. JLGC conceived the study, supervised analyses, and revised the manuscript. EP and AP analyzed the data. SS, NT, and CT conceived and designed the study cohort. Also see the “Acknowledgments” section.
None declared.