This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.
Effective interventions aimed at correcting COVID-19 vaccine misinformation, known as fact-checking messages, are needed to combat the mounting antivaccine infodemic and alleviate vaccine hesitancy.
This work investigates (1) the changes in the public's attitude toward COVID-19 vaccines over time, (2) the effectiveness of COVID-19 vaccine fact-checking information on social media engagement and attitude change, and (3) the emotional and linguistic features of the COVID-19 vaccine fact-checking information ecosystem.
We collected a data set of 12,553 COVID-19 vaccine fact-checking Facebook posts and their associated comments (N=122,362) from January 2020 to March 2022 and conducted a series of natural language processing and statistical analyses to investigate trends in public attitude toward the vaccine in COVID-19 vaccine fact-checking posts and comments, and emotional and linguistic features of the COVID-19 fact-checking information ecosystem.
The percentage of fact-checking posts relative to all COVID-19 vaccine posts peaked in May 2020 and then steadily decreased as the pandemic progressed (
The percentage of fact-checking posts relative to all posts about the vaccine steadily decreased after May 2020. As the pandemic progressed, third-party fact checkers played a larger role in posting fact-checking COVID-19 vaccine posts. COVID-19 vaccine fact-checking posts continued to be more analytical and more confident over time, reflecting increased confidence in posts. Similarly, tentativeness in comments decreased; this likewise suggests that public uncertainty diminished over time. COVID-19 fact-checking vaccine posts from hospitals yielded more positive attitudes toward vaccination than other information sources. At the same time, hospitals received less engagement than other information sources. This suggests that hospitals should invest more in generating engaging public health campaigns on social media.
As of May 4, 2022, the novel COVID-19 outbreak had caused 994,551 deaths and 81,574,159 cases in the United States [
Thus, effectively communicating the necessity of getting the COVID-19 vaccine is essential to mitigating the COVID-19 pandemic. Although some public officials have endorsed the COVID-19 vaccine, others have fostered vaccine hesitancy by broadcasting misinformation (ie, inaccurate health information), which is often disseminated widely via social media [
Research efforts have therefore focused on experimentally testing the efficacy of COVID-19 vaccine misinformation fact-checking messages, finding that accurate misinformation correction messages can effectively mitigate health misinformation in certain contexts, namely when the message is from a credible information source (ie, health institutions, research institutions, and news media) [
Vaccine attitude determines people’s intention to vaccinate and the consequential vaccine uptake behaviors [
In addition, it is also important to investigate the attitude toward COVID-19 vaccine fact-checking messages itself. Emotionally charged messages are found to influence vaccination intent more than facts and statistics [
We know politicians are a prevalent source of vaccine misinformation on social media (Featherstone et al. [
In addition to the public's valenced attitude toward the COVID-19 vaccine, varied discrete emotions may reveal more about the specific attitudes or concerns. Different discrete emotions have different effects on the vaccine-hesitant. For example, vaccine-hesitant users are more likely than provaccine users to express anger in posts and replies [
Although staunch antivaxxers exist, a prominent group of Americans who understand vaccine importance are hesitant to take it because of uncertainty about the safety of the vaccine’s rushed development [
To fill these gaps in the literature, we collected a data set of 12,553 COVID-19 vaccine fact-checking posts and their associated comments (N=122,362) from January 2020 to March 2022. Facebook was selected because it is 1 of the most popular social media platforms worldwide with a significant presence of both misinformation and fact-checking information [
We leveraged Meta’s
CrowdTangle tags public Facebook pages based on several attributes, including the primary language of the content, the country the content is geared toward, and the type of entity that owns the page (eg, health influencer, top newspaper). We curated a list of pages belonging to categories related to health information sharing, namely third-party fact checkers, general media sources, top newspaper sources, health influencers, health media, hospitals, and wellness publications (
Second, we mined the total number of COVID-19 vaccine–related posts (N=151,008) and the accompanying post metadata provided with Crowdtangle (ie, date, number of shares, comments, and emoticon reactions) from the curated list of health-related Facebook pages (N=2644) [
Further, we extrapolated posts containing at least 1 (8%) of 13 fact-checking keywords (eg, “debunk,” “hoax”) to distill fact-checking posts (N=12,553) from the larger data set (
Process to identify relevant lists of Facebook pages, collect Facebook posts and comments, and filter relevant Facebook posts.
Although CrowdTangle makes some metadata available (ie, post date, number of shares, and number of Facebook emoticon reactions), it does not provide access to comment data. We used an automatic data collection software called
We used Google Cloud Natural Language AI, a machine learning-based natural language–understanding tool to retrieve the public's attitude toward the COVID-19 vaccine by (1) identifying all entities that were discussed in a given post or comment, (2) using COVID-19 vaccine keywords to distill entities specifically related to the COVID-19 vaccine (eg, COVID-19 vaccine, Pfizer Booster, etc; see Table S3 in
Specifically, we focused on 3 dimensions of the attitude: COVID-19 vaccine entity salience (ie, the salience of COVID-19 vaccine–related entities of all entities in a given text), attitudinal valence (ie, the positive or negative attitude toward vaccine-related entities), and attitudinal magnitude of each COVID-19 vaccine entity (ie, how strong the attitude is).
COVID-19 vaccine
COVID-19 vaccine
COVID-19 vaccine
Inferences can be made regarding attitude by combining these 2 dimensions. For example, a comment thread that yields a close-to-zero attitudinal valence and close-to-zero attitudinal magnitude indicates that the thread is unemotional. However, a comment thread that yields a close-to-zero attitudinal valence but high attitudinal magnitude suggests the existence of both highly positive and highly negatively valenced attitudes in the comments that cancel each other out [
Social media engagement was operationalized as the number of comments and shares each post received, as well as the number of Facebook reactions a given post received. Facebook reactions are a series of 6 emoticons that enable users to express their emotional responses to posts [
Facebook reaction emoticons.
We used IBM Watson Tone Analyzer, a classifier of discrete emotions and linguistic features based on cognitive linguistic analysis to extract discrete emotions and linguistic features. In contrast to Google Cloud Natural Language AI [
This study did not involve human subjects and therefore did not need an institutional review board (IRB) review. The data involved was public data with no identifiable information [
To answer RQ1 on the changes of the public's attitude toward the COVID-19 vaccine, we aggregated the COVID-19 vaccine fact-checking posts by time and conducted a series of correlation tests between the attitude variables (ie, vaccine entity salience, attitudinal valence, and magnitude) and time.
To answer RQ2 on the effects of different COVID-19 vaccine fact-checking information sources, we conducted multiple linear regressions with the public's attitude toward the COVID-19 vaccine as the dependent variable and negative binomial regressions with social media engagement as the dependent variable. In these regression models, the word count in the posts and comments, discrete emotions and linguistic features in the posts, and Facebook page followers were controlled.
To answer RQ3 and RQ4 on the changes in discrete emotions and linguistic features over time, we conducted a series of correlation tests between discrete emotions, linguistic features, and time.
The number and percentage of COVID-19 fact-checking posts are shown in
Interestingly, we found that the 2 peaks of the COVID-19 vaccine posts corresponded with the key time points of COVID-19 vaccination (
Number of COVID-19 vaccine fact-checking Facebook posts compared to all COVID-19 vaccine–related posts aggregated by month, from January 1, 2020, to March 10, 2022. The line represents the number of new COVID-19 cases, aggregated by month, from January 1, 2020, to March 10, 2022.
Changes in the percentage of COVID-19 vaccine fact-checking Facebook posts in all COVID-19 vaccine–related Facebook posts over time, aggregated by month, from January 1, 2020, to March 10, 2022.
Vaccination population in the United States over time: the number of individuals receiving at least 1 dose, the number of fully vaccinated individuals, and the number of booster shots issued, aggregated by month, from January 1, 2020, to March 10, 2022.
To answer RQ1, we aggregated vaccine entity salience and attitudinal valence and magnitude by month. The salience of COVID-19 vaccine–related entities relative to off-topic entities is depicted in
The mean of COVID-19 vaccine attitudinal valence in posts (mean 0.01, SD 0.10) and comments (mean –0.004, SD 0.12) was close to 0. Since the average attitudinal magnitude of posts (mean 0.05, SD 0.11) and comments (mean 0.09, SD 0.13) was close to 0, results revealed that the texts were relatively neutral. Notably, COVID-19 vaccine attitudinal magnitude in comments increased during the pandemic (
Therefore, we noticed a decrease in vaccine entity salience in posts over time (RQ1a) and an increase in attitudinal magnitude in comments over time (RQ1b). Attitudinal valence did not change significantly over time for both posts and comments.
Salience of COVID-19 vaccine–related entities over time, aggregated by month, from January 1, 2020, to March 10, 2022.
COVID-19 vaccine attitudinal valence and magnitude in COVID-19 vaccine fact-checking posts and comments aggregated by month, from January 1, 2020, to March 10, 2022.
The majority of COVID-19 vaccine fact-checking Facebook posts were generated by news media (n=5821, 46.4%) and hospitals (n=4921, 39.2%), while relatively fewer posts were posted by third-party fact checkers (n=1523, 12.1%) and US health media (n=288, 2.3%); see
Social media engagement was operationalized as 9 different metrics, namely, the number of comments (mean 146.76, SD 561.07, median 9) and shares each post received (mean 92.28, SD 594.93, median 8), as well as the number of Facebook reactions a given post received (see
Health media have more followers than other 3 sources, while hospitals on average have the least number of followers. COVID-19 vaccine fact-checking posts created by news media were most popular, with the highest number of likes, comments, and shares on average, and posts by hospitals were least popular.
Summary statistics for sources of COVID-19 fact-checking posts.
Information source | Facebook pages (N=2644), n (%) | Posts (N=12,553), n (%) |
News media | 95 (3.6) | 5821 (46.4) |
Hospitals | 1096 (41.5) | 4921 (39.2) |
Third-party fact checkers | 9 (0.3) | 1523 (12.1) |
US health media | 26 (1.0) | 288 (2.3) |
Percentage of COVID-19 vaccine fact-checking Facebook posts by 4 information sources in the United States, aggregated by month, from January 1, 2020, to March 10, 2022.
Summary statistics for Facebook reactions.
Reaction metric | Mean (SD) | Median |
Like | 427.65 (1946.95) | 41 |
Love | 42 (346.86) | 1 |
Wow | 26.04 (199.94) | 0 |
Haha | 64.96 (406) | 2 |
Sad | 26.95 (218.49) | 0 |
Angry | 76.33 (554.58) | 0 |
Care | 3.96 (19.84) | 0 |
Average social media engagement of COVID-19 vaccine fact-checking Facebook posts across information sources.
Category | Third-party fact checker | Health media | Hospital | News media |
Average followers | 538,028 | 2,193,198 | 15,639 | 6,619,852 |
Posts (N=12,553), n (%) | 1523 (12.1) | 288 (2.3) | 4921 (39.2) | 5821 (46.4) |
Likes, mean (SD) | 231.55 (587.35) | 252.07 (1039.86) | 24 (162.96) | 828.88 (2,774.68) |
Comments, mean (SD) | 125.1 (228.83) | 144.38 (320) | 9.39 (150.71) | 268.67 (781.04) |
Shares, mean (SD) | 100.14 (476.54) | 125.02 (398.62) | 27.07 (650.73) | 143.72 (576.06) |
Love, mean (SD) | 7.66 (58.81) | 10.77 (31.86) | 3.34 (33.04) | 85.22 (504.1) |
Wow, mean (SD) | 9.7 (33.28) | 10.3 (44.35) | 0.16 (1.62) | 52.96 (290.61) |
Haha, mean (SD) | 50.08 (184.47) | 33.14 (89.59) | 1.32 (42.64) | 124.23 (581.05) |
Sad, mean (SD) | 10.67 (42.26) | 8.4 (38.94) | 0.5 (6.86) | 54.49 (317.71) |
Angry, mean (SD) | 43.56 (194.11) | 10.24 (32.22) | 0.51 (10.83) | 152.27 (801.37) |
Care, mean (SD) | 1.2 (5.91) | 2.75 (10.14) | 0.98 (11.66) | 7.26 (26.45) |
To answer RQ2a, we conducted multiple linear regressions to investigate the effects of information sources on COVID-19 vaccine attitude, with word counts in the posts and comments and the number of Facebook page followers included as control variables (
To answer RQ2b, we aggregated fact-checking posts by Facebook page information source and created negative binomial regression models to assess whether the type of information source significantly predicted the 9 metrics of social media engagement, while controlling for post word count and follower count (
Results revealed that hospitals have a significantly lower social media engagement for all engagement metrics than news media. Similarly, health media had significantly fewer wow and angry reactions than news media (
Regression coefficients (95% CI) of information sources with significant effects on social media engagement in negative binomial models. Blue dots and blue error bars show significant coefficients and 95% CIs (
To answer RQ3, we used IBM Watson Tone Analyzer to extract 4 discrete emotions and 3 linguistic features in COVID-19 vaccine fact-checking posts and comments (
Trends in discrete emotions in COVID-19 vaccine fact-checking posts and comments, aggregated by month, from January 1, 2020, to March 10, 2022.
To answer RQ4, we used IBM Watson Tone Analyzer to extract 3 linguistic feature attributes of COVID-19 vaccine fact-checking posts and comments, namely post analytical thinking, confidence, and tentativeness (
In addition, linguistic features changed over time (
Although comments did not exhibit a significant increase in confidence over time, tentativeness in comments decreased significantly (
Changes in linguistic features in COVID-19 vaccine fact-checking posts and comments, aggregated by month, from January 1, 2020, to March 10, 2022.
In addition to the proposed RQs, we explored how comments might respond differently across information sources regarding discrete emotions and linguistic features with multiple linear regression analyses. The word count of posts and comments, the Facebook page follower count, and discrete emotions and linguistic features of posts were controlled (
Results revealed that fact-checking posts from hospitals were associated with lower levels of anger, fear, and sadness in comments, while posts from third-party fact checkers were associated with heightened comment anger. In other words, third party fact checkers tended to evoke heightened comment anger, whereas comments on posts from health media and hospitals expressed less negative emotion. However, no significant effects were found for linguistic features.
Regression coefficients (95% CIs) of information sources with significant effects on emotions and language tones in comments in linear regression models. Blue dots and blue error bars show significant coefficients and 95% CIs (
This study examined the US COVID-19 vaccine fact-checking information on Facebook and analyzed the effects of different fact-checking posts’ information sources on the public's attitude toward COVID-19 vaccines and social media engagement. We observed the prevalence and trend of COVID-19 vaccine fact-checking information on Facebook. Findings revealed health information Facebook pages responded to the COVID-19 infodemic by posting most frequently at 2 key vaccine time points in the United States: (1) when the vaccine first became available in December 2020 and (2) when the booster shot became available in August 2021 [
Notably, the percentage of fact-checking posts relative to all COVID-19 vaccine posts steadily decreased as the pandemic progressed. This may be because the frequency of COVID-19 vaccine posts increased at a higher rate than fact-checking vaccine posts. Another explanation is that public health organizations’ efforts to promote accurate COVID-19 vaccine information reduced COVID-19 vaccine misinformation, necessitating that users fact-check misinformation. Likewise, posts tended to focus more on COVID-19 vaccine entities than comments did, reflecting the public concern over a more diverse set of topics relative to the vaccine itself. This may be because as the pandemic progressed, Facebook and sources of health information took actions to mitigate vaccine misinformation, necessitating less misinformation corrections and vaccine discourse over time. For example, Facebook began removing COVID-19 health misinformation and attaching various warnings to misleading posts, and public health initiatives promoted accurate COVID-19 vaccine information [
Our most prominent finding was that hospitals play a key role in disseminating facts and correcting misinformation. Although hospitals receive less engagement than other information sources, the comments expressed more positive emotions compared to other information sources. This suggests that hospitals should invest more in generating engaging public health campaigns on social media.
Regarding overall emotions in the comments, fact-checking posts from health media and hospitals were associated with lower levels of anger, fear, and sadness in the comments, while posts from third-party fact checkers were associated with higher levels of anger in the comments. These negative emotions are crucial heuristic cues to the public's attitude and therefore should be acknowledged by information and health influencers in communicating facts and correcting misinformation. Empathetic communication enables fact-checking practitioners to better connect with the audience and counterbalance the negative emotions and hesitancy evoked by COVID-19 vaccine misinformation [
In addition, the majority of COVID-19 vaccine fact-checking Facebook posts were generated by news media and hospitals, while relatively few were from third-party fact checkers and US health media. Notably, third-party fact checkers posted more COVID-19 vaccine posts as the pandemic progressed. Although health media were the smallest source COVID-19 vaccine fact-checking posts, they have more followers than the other 3 sources, and although hospitals generated more fact-checking posts, they have the fewest followers. Although health media posts had similar levels of engagement as news media, they elicited few wow and angry reactions, likely reflecting a less negative attitude amongst followers of health media compared to news media. This may be because news media communicate with the general public, while health-concerned people follow health media and tend to have consistent health views. COVID-19 vaccine fact-checking posts created by news media were most popular, with the highest number of likes, comments, and shares on average, whereas users engaged with posts from hospitals the least.
Posts and comments tended to be relatively neutral in nature with low levels of attitudinal valence. However, as the pandemic progressed, the salience of COVID-19 vaccine entities in posts kept decreasing, and the publics’ comments became more extreme, with higher levels of attitudinal magnitude. This suggests that fact-checking posts tend to report news and communicate facts objectively and have shifted the focus from the COVID-19 vaccine itself to other related subjects. However, the public’s attitude became increasingly extreme over time. This supports extant findings that early interventions, such as inoculation against misinformation before attitude becomes increasingly extreme, may be more effective in the long term [
In addition, the salience of COVID-19 vaccine entities was significantly lower in comments than in posts. This suggested that the public is more concerned with issues other than the COVID-19 vaccine. The discrepancy in posts and comments further suggests the need for responsive and empathetic communication that might be more effective in improving the vaccine attitude and confidence.
In line with our conclusion that public comments became more extreme as the pandemic progressed, fact-checking comments exhibited heightened joy, anger, fear, and sadness than posts. Although the presence of heightened positive emotions (eg, joy) in COVID-19 health messages has shown to predict compliance with COVID-19 public health guidelines [
Likewise, users may seek and engage with sad content to manage negative emotions [
Emotions and linguistic features in both COVID-19 vaccine fact-checking posts and comments evolved over time. The posts adopted a more analytical and confident tone over time, while we observed a significant drop in fear and tentativeness in the comments. Both trends suggest that with more information we know about the pandemic and the COVID-19 vaccine, the confidence related to the COVID-19 vaccine increases for both information sources and the general public.
Although our findings shed light on COVID-19 vaccine fact-checking in a naturalistic setting, this study is not without limitations. By focusing on a sample of US posts, we neglected to explore how fact-checking manifests in other countries. Additionally, as different platforms have different behavioral norms [
This study has broad implications for public health practitioners and social media managers. First, although hospitals play a large role in fact-checking COVID-19 vaccine misinformation, they should work to design posts that will better engage the public. Hospitals are perceived as credible information sources with authority as health institutions, which makes them credible sources that are highly likely to elicit attitude and behavior change on health issues. Second, as fact-checking posts evoked increasingly extreme public attitude over time, early interventions (ie, social media campaigns that inoculate against misinformation before it becomes mainstream) are critical. Additionally, fact-checking information sources should engage in empathetic communication to better address the concerns of the public and empathize with the public. For example, sadness can both protect against belief in misinformation and facilitate attitude change when confronted with misinformation corrections [
Supplemental material.
artificial intelligence
research question
None declared.