We conducted a pilot study using a pretest-posttest design to evaluate “Vac Chat, Fact Check,” a chatbot designed to provide updated information and debunk misinformation about the COVID-19 vaccine. The study was conducted in November 2021, between the end of the fourth wave (June 2021) and the start of the fifth wave (January 2022) of the outbreak in Hong Kong, which had about 12,000 cumulative cases and 200 deaths.

The study targeted adults aged ≥18 years who can read and communicate in Chinese. A mass email with a link to a web-based survey of COVID-19 vaccination was sent to all students at a public university in Hong Kong on November 8, 2021. The survey link was open for 7 days and received 290 valid responses. Of these, 273 (94.1%) respondents indicated their interest in participating in the pre-post evaluation of the chatbot by leaving their contact information at the end of the survey. We identified and invited all 46 respondents who were either unvaccinated (n=22) or fully vaccinated but hesitant to receive boosters if eligible (n=24; response rate: 46/46, 100%). The planned sample size (20-25 each for unvaccinated or booster-hesitant participants) was based on a previous formative study of a chatbot for promoting human papillomavirus vaccination [18]. Figure 1 shows the study flow diagram.

The study was approved by the Institutional Review Board of the University of Hong Kong/Hospital Authority Hong Kong West Cluster (UW 21-449).

Participants who were invited to participate in the pre-post study received a WhatsApp message describing the study purpose and provided informed consent. The participants then received a URL link to access the web-based chatbot and start a 7-day trial period. The chatbot could be accessed repeatedly. WhatsApp reminders to use the chatbot were sent on day 3, day 5, and day 7. On day 8, we sent a URL link to the postintervention questionnaire. Participants who completed the pre-post study were given HK $300 (US $38.5) for their time and effort.

On March 30, 2022, about 4 months after the completion of the pre-post study, we conducted a post hoc follow-up with a single question about COVID-19 vaccination or booster status via WhatsApp. Participation was voluntary, and consent was obtained from those who responded to the question. The additional follow-up served to examine the feasibility of measuring the long-term effect of the chatbot.

The “Vac Chat, Fact Check” chatbot was developed by our team. To promote dissemination, the chatbot could be accessed by any internet browser on smartphones, tablets, and personal computers (ie, web-based). The chatbot was available in Chinese only since most Hong Kong residents (>90%) spoke Chinese. Upon entering the chatbot, the user received a message about how to use the chatbot and a menu of options showing the core functionality of the chatbot. The users could navigate the chatbot by typing the corresponding number of the options in the menus or keywords (eg, allergy) to obtain information directly (Figures 2 and 3).

The chatbot conversation generally unfolded by following predefined rules or decision trees. To better simulate human interactions, the chatbot also used natural language processing (NLP) powered by Google Dialogflow for handling small talk (eg, greetings and thank-yous). The chatbot provided responses mainly using texts with emojis, but some messages also included infographics (Figure 3).

The intervention content followed the Confidence, Complacency and Convenience (“3C’s”) model of vaccine hesitancy [19]. Specifically, the information addressed the lack of trust in the effectiveness and safety of the vaccine (confidence), the lack of perceived risk of COVID-19 or the perception that vaccination is not necessary (complacency), and barriers to access the vaccine (convenience). The information provided by the chatbot was categorized into 6 major topics (Table 1). Our population-based survey suggested that inadequate knowledge about COVID-19 could contribute to vaccine hesitancy [16]. Therefore, the chatbot included general information about COVID-19. Since 2 types of vaccine (inactivated and mRNA vaccines) were available in Hong Kong with differing eligibility criteria (age and pregnancy status), the chatbot also included a decision aid for selecting the suitable vaccine.

To evaluate the chatbot efficacy, we used 1-sample, 2-tailed t test and Wilcoxon sign-rank test to examine the change in intention to receive a vaccine or booster and COVID-19 VHS scores from preintervention to postintervention. We also examined changes in the “Lack of confidence” and “Risk” subscales of the COVID-19 VHS. The effect size of the pre-post difference in COVID-19 VHS scores (Cohen d) was calculated as a mean difference divided by the SD of the mean difference. Our sample size of 46 participants could detect a moderate effect size of 0.43 (Cohen d) in the pre-post difference in COVID-19 VHS scores with 80% power at 2-sided 5% level of significance. The corresponding effect sizes detectable were 0.64 for intention to vaccinate (n=21) and 0.60 for intention to receive boosters (n=24). Intervention usability and acceptability were reported descriptively. For the secondary aim, we used bivariate and multivariable linear regression to examine the factors associated with the COVID-19 VHS score in all survey respondents. Factors examined included sociodemographic characteristics, chronic disease status, previous flu vaccination, eHealth literacy, and the perceived susceptibility to and severity of COVID-19.

All statistical analyses were conducted in Stata/MP software (version 15.1; StataCorp). We used complete case analyses because there were no missing data in the web-based survey and postintervention assessment except eHealth literacy (n=2) and the perceived susceptibility (n=4) and severity (n=4) of COVID-19. A 2-sided P<.05 denoted statistical significance.

The mean (SD) age of all survey respondents was 21.4 (6.3) years, and 61% (177/290) of respondents were female (Table 2). Participants of the pre-post study (n=46) had similar characteristics to those of nonparticipants (n=244) except, as expected, having significantly higher COVID-19 vaccine hesitancy (P<.001).

In all survey respondents, both bivariate and multivariable models showed that lower eHealth literacy and perceived danger of COVID-19 were associated with higher COVID-19 vaccine hesitancy (Table 3). The results were similar after additionally adjusting for COVID-19 vaccination status (data not shown).

The completion rate of the postintervention assessment was 100% (46/46). Table 4 shows the favorable changes in all measures related to COVID-19 vaccination from preintervention to postintervention (mean duration: 15.0 days). The main efficacy outcome of COVID-19 VHS score significantly decreased from 28.6 (preintervention) to 24.5 (postintervention), with a mean difference of –4.2 (P<.001) and an effect size (Cohen d) of 0.94. Similarly, both the “Lack of confidence” and “Risk” subscale scores significantly decreased. Intention to vaccinate or receive boosters and willingness to encourage others to vaccinate significantly increased from preintervention to postintervention. One unvaccinated participant at preintervention reported having received the first dose of the vaccine at postintervention.

On average, the participants used the chatbot for a total of 64 (SD 47) minutes during the 1-week trial period. Longer time spent on the chatbot was correlated with a larger reduction in vaccine hesitancy with marginal significance (Spearman ρ=0.26; P=.08). Among participants who used the chatbot (n=46), the median (IQR) SUS score was 72.5 (65-77.5) out of 100. On a scale from 1 (not agree at all) to 5 (strongly agree), the median (IQR) score on the perceived usefulness of the chatbot was 4 (4-4) for getting information about the COVID-19 vaccine, 3 (2-4) for making decisions about vaccination, and 3 (2-3) for increasing the motivation to get vaccinated. The median (IQR) recommendation score was 7 (6-8) on a scale from 0 to 10.

Overall, 18 (82%) of the 22 initially unvaccinated participants and 19 (79%) of the 24 booster-hesitant participants responded to the post hoc 4-month follow-up. All 18 unvaccinated participants reported having received COVID-19 vaccination (2 doses: n=16, 89%; and 1 dose: n=2, 11%), whereas 7 (37%) of the 19 booster-hesitant participants reported having received boosters.

This pilot study showed a significant decrease in COVID-19 vaccine hesitancy after using the “Vac Chat, Fact Check” chatbot in young adults who were hesitant to vaccinate or receive boosters. According to the rule of thumb of Cohen [29], the effect size (Cohen d=0.94) was large. Other efficacy outcomes, including intention to vaccinate or receive boosters and willingness to encourage others to vaccinate, consistently showed the benefit of the chatbot. The usability of the chatbot was supported by the median SUS score of 72.5 out of 100, which fell between the cutoffs of “good” (a score of 71.4) and “excellent” (a score of 85.5) adjective ratings [30]. The median recommendation score of 7 on a scale from 0 to 10 indicated the satisfactory acceptability of the chatbot [31].

Our PubMed search using the keywords of vaccine and chatbot and their synonyms only identified 1 peer-reviewed study that provided empirical evidence on the efficacy of a chatbot for promoting COVID-19 vaccination. The study was a web-based experiment on a French sample population, which found that interacting with a chatbot could promote more positive attitudes toward COVID-19 vaccines and intention to vaccinate [32]. A study (preprint) also showed an increase in vaccine acceptance in Japanese adults after using “Corowa-kun,” a chatbot in LINE instant messenger [33]. Direct comparison between our study with these studies were difficult because of differences in the study methods, sample characteristics, and outcome measures. Nevertheless, our findings were consistent with these studies by showing a positive impact of chatbot on COVID-19 vaccine uptake.

To our knowledge, our study was the first to include actual receipt of COVID-19 vaccines or boosters as outcome measures in chatbot evaluation. Assuming (conservatively) that all participants lost to follow-up did not receive any vaccine or booster, 82% (18/22) of the initially unvaccinated participants received at least 1 dose of vaccine, whereas 29% (7/24) of booster-hesitant participants received a booster. As a reference, the corresponding rates were 92% and 28% in Hong Kong residents aged 20 to 29 years on March 30, 2022 (same date as the follow-up) [34]. Note that these figures could not be directly compared because of differences in sample characteristics, and our participants were likely more vaccine- or booster-hesitant than the general population. Nonetheless, the satisfactory response rate of 80% (37/46) provides support for the feasibility of conducting longer-term (>3 months) follow-up in future trials.

Corroborating our previous findings in the general population [16], we found that higher perceived severity of, but not susceptibility to, COVID-19 was associated with lower COVID-19 vaccine hesitancy. Previously studies have found that eHealth literacy was associated with knowledge and adherence to nonpharmacological preventive measures against COVID-19 [17,35]. This study further found eHealth literacy to be associated with COVID-19 vaccine hesitancy. Higher eHealth literacy helps people process and discern the credibility of web-based health information, which may buffer the impact of the infodemic (an overabundance of information, both accurate or otherwise, during a disease outbreak) and misinformation against the vaccine and thus hesitancy. Our findings corroborate the importance of building eHealth literacy to fight the COVID-19 pandemic.

Similar to most chatbots built to support health care amid the COVID-19 pandemic [11,12], our “Vac Chat, Fact Check” chatbot was primarily rule-based. We decided against building a chatbot that is entirely powered by NLP for practical reasons. First, existing and readily available NLP engines remain inadequate in handling free-flow conversations in Cantonese (the local Chinese dialect). Second, rule-based chatbots are relatively inexpensive and could be quickly developed and deployed to mitigate the pandemic when health care resources are stretched. NLP-based chatbots could better simulate human interaction but require extensive training and resources to become adequately usable. Nonetheless, our study has provided proof-of-concept evidence to support chatbots as a mode of delivery to promote vaccination, which provides the impetus for developing more sophisticated and potentially more effective chatbots.

The main limitation of the pre-post study is the lack of a control or comparison group, which limited the causal inference of any changes observed after using the chatbot. The possibility that the observed changes were attributable to contextual changes along the course of the outbreak could not be excluded. However, the study was conducted at a time when Hong Kong had been maintaining a low level of local transmission with nearly 0 daily local case (from June to December 2021). This setting, coupled with the short interval between preintervention and postintervention assessments, was unlikely to have had a substantial effect on the vaccination outcomes. Nevertheless, the findings must be considered preliminary and hypothesis-generating. Another limitation is the small sample size, which precludes the examination of the chatbot’s efficacy in sociodemographic subgroups (eg, sex). Third, since all measures were self-reported, social desirability bias could not be excluded. Finally, our study targeted young adults given their greater vaccine hesitancy than older populations and their frequent use of social networking sites—a major source of misinformation. The generalizability of the findings to other populations is unclear. Due to the convenience sampling method, our participants may not be representative of all young adults who are unvaccinated or booster-hesitant.

Promoting the uptake of COVID-19 vaccines is crucial to mitigating the impact of COVID-19. This pilot study provided initial evidence to support the efficacy, usability, and acceptability of a chatbot for promoting COVID-19 vaccination in young adults who were unvaccinated or booster-hesitant. Randomized controlled trials are warranted to test the effectiveness of the chatbot in increasing COVID-19 vaccination. Although our study indicated the benefits of the chatbot in both unvaccinated and booster-hesitant young adults, the drivers for vaccine hesitancy between the 2 groups likely differ. Further research is also needed to understand their differences to provide more tailored information and optimize the chatbot’s efficacy.