On March 11, 2020, the World Health Organization (WHO) declared the COVID-19 outbreak a pandemic [1]. Since then, the virus has had global impacts, and as of July 2021, there had been over 185 million confirmed cases of COVID-19 and over 4 million deaths [2]. The virus has at times overwhelmed the health systems of different countries, and the need for accurate and timely information has never been higher. The COVID-19 pandemic has seen users around the world turn to digital technology for the sourcing of health information [3]. Health authorities such as WHO have responded with new and innovative ways of delivering trusted information, including the use of chatbots [4]. Chatbots are software systems that enable users to interact with a program as if they are talking to another person, often using machine learning to achieve the effect of intelligent response. Chatbots are being used across all areas of health to deliver information, promote behavior change, and deliver treatment. Chatbots can be deployed as stand-alone systems or via existing communication platforms such as WhatsApp or Facebook Messenger. This integration into familiar apps can result in a low barrier to entry for many users and a potential for substantial reach.

Chatbots can be rule-based or artificial intelligence–based. Rule-based chatbots use decision trees and defined rules to guide conversations, whereas artificial intelligence–based chatbots uses machine learning and natural language processing to generate and respond to dialogue [5]. As chatbots become more sophisticated, this opens up new and innovative ways to interact with and engage users in digital content. Features such as videos, quizzes, emoji, and style of voice are all used to keep users engaged. As these tools are used more, it is important to understand the marketing and communication strategies used to improve effectiveness and user experience. This enables the design of systems that people want to use and can benefit from.

Good user experience design is important for chatbot adoption and acceptance, and this requires expert input [6]. Digital solutions that are not designed with the user in mind can result in low engagement [7]. Working in a co-design model and involving end users throughout development can help in developing user-centered products and increasing engagement [7]. Chatbots have been developed for a wide range of health issues and behaviors, with mental health chatbots having the most dedicated literature [8]. A recent scoping review of user perspectives and opinions about mental health chatbots found that people generally found them easy to use and that enjoyment and trust are key mediators of interaction with chatbots [9]. However, the quality of conversation was identified as a limitation.

A review of technical metrics used to evaluate chatbots found a diversity in approaches and no apparent standardization [8]. The authors identified 27 metrics related to chatbots as a whole, response generation, response understanding, and aesthetics. They identified a range of ways to measure usability, such as a single question in a questionnaire, multiple questions, observation, or the use of a validated scale such as the System Usability Scale. In addition, they found that only 7% of studies included any assessment of aesthetics [8]. A scoping review of the features most commonly used in mental health chatbots showed that most were rule-based and stand-alone software [10]. In most chatbots, the conversations were led by the chatbot, and most included digital representations such as an avatar or digital human characters [10].

Chatbots can have special use in a pandemic for reaching people with information, supporting behavior change, providing mental health support, and identifying and monitoring symptoms [11]. Recognizing these opportunities, chatbots for COVID-19 have been quickly developed and scaled up by health authorities such as the WHO [12] and Centers for Disease Control and Prevention [13]. COVID-19 has overwhelmed health systems around the world [14], and the combination of increased need for health services and the need for social distancing has highlighted the potential for chatbots to ease some of the health system burden [15]. A number of studies describe guidance or frameworks for the design of chatbots for COVID-19 [15-17]. Despite the importance of assessing adoption and user engagement, they are not routinely reported in chatbot evaluations, and measures are not standardized [3,4]. The COVID-19 pandemic has presented a unique situation that has accelerated the need for the implementation of digital solutions. Although scoping reviews have previously examined COVID-19 chatbots in general [18], this is the first paper to our knowledge that focuses on this specific scenario. This paper sought to describe the current knowledge base about the user experience of COVID-19 chatbots.

We conducted a scoping review of the user experience of COVID-19–related chatbots, with regard to design, engagement, and communication features. We followed guidance from the Joanna Briggs Institute on conducting systematic scoping reviews [19] and reported according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines for scoping reviews [20]. This guidance outlines the steps, from the development of a protocol and the search strategy to charting the results and reporting the findings.

We sought to provide an overview of the evidence of design, engagement, and communication features used in COVID-19 chatbots and the impact on user engagement, preference, and retention.

Our review questions were as follows:

We used a standard 3-step search strategy for scoping reviews.

For this review, we searched databases from June 19-20, 2021. The search strategy was initially developed for MEDLINE and then adapted for other databases. We initially searched MEDLINE Ovid and Embase. Search results from these databases were reviewed, and the search strategy was adjusted slightly before being rerun. Other databases used in the search included CINAHL, ACM Digital Library, Cochrane COVID-19 study register, Emerald, Communication abstracts (EBSCO), and the WHO COVID-19 Global literature on coronavirus disease. To source unpublished literature, Google Scholar was searched using the adapted search terms and the first 100 results were scanned. Backward and forward citation searches of all the included articles were conducted, and additional articles identified were sourced individually.

The search terms for this review were first developed with previous knowledge from the authors and then further informed by the current literature about this topic. A comprehensive list of search terms was developed to ensure a broad range of articles. We used search terms related to the technological subject matter of interest (eg, chatbots, conversational agent, and dialogue system), user experience more generally (eg, engagement, features, and user experience), and the health issue at hand (eg, COVID, COVID-19, Corona, and coronavirus). The search terms used for each database can be viewed in Multimedia Appendix 1.

We included studies that assessed the design, marketing, and user features of COVID-19 chatbots or those that explored user perspectives on features, marketing, and design. As the aim was to map the existing evidence across published and unpublished literature, there was no criteria limitation with regard to geographical location, type of chatbots, or study design. We included papers, articles, and conference proceedings. Any articles or reports pertaining to chatbots that were not related to COVID-19 were excluded. We excluded papers that did not include any reporting on user perspectives, experience, or the general use of chatbot features or marketing, as well as papers where a version was not available in English. We excluded conference abstracts, books, and theses.

The selection of articles was defined by the inclusion and exclusion criteria. In line with the iterative nature of scoping reviews, these criteria were refined as the review progressed with alterations made to the criteria following the initial search of 2 databases. Deduplication was carried out in Endnote software (Clarivate), and then Rayyan was used for screening. Rayyan is a web-based program designed for managing the citation-screening process [21]. Authors BW and AM independently screened article titles and abstracts, excluding irrelevant studies. Any discrepancies were resolved via discussion. For articles deemed possibly relevant or where it was difficult to ascertain from the title and abstract, the full text was reviewed. Figure 1 shows the study selection process.

A data extraction form was developed to ensure the uniform collection of data, and this form was trialed and revised. Data were charted in categories of study characteristics, intervention characteristics, and outcomes. A thematic analysis was then used to synthesize the findings in each paper and identify themes.

Of the 517 papers initially identified, 10 were included in the final analysis, with 2 papers reporting results from the same study population. The authors took a wide view of the inclusion criteria, including studies that included any reporting of the abovementioned criteria. The papers differed widely in their approach to reporting user experience, from empirical studies that evaluated user experience comprehensively to those that reported minimal findings. Several papers were excluded even though they had detailed their use experience plan because they did not report findings. Papers included discussions about chatbots developed for COVID-19 screening and general COVID-19 information, as well as studies investigating user perceptions and opinions on COVID-19 chatbots. A thematic analysis was carried out on the included studies to understand and organize the findings across the papers. Table 1 describes the characteristics of the included studies.

The papers described a range of factors impacting on user experience and chatbot use. The COVID-19 pandemic expedited the development of chatbots with some tools being developed and deployed within 5 days [25,27]. Partnership with technical teams was described as being beneficial [25], with one paper describing the lack of technical expertise on their team as a limitation [30]. Findings are mapped by theme and described in the following section.

This paper describes what is known about the user experience and user uptake of COVID-19 chatbots. The 10 papers in this review included some describing chatbots for COVID-19 screening, some describing chatbots for information dissemination, and others seeking to understand uptake and user perspectives. The papers generally reported good acceptance of chatbots and reported a number of factors that appeared to determine this. Key themes included content, trust, digital ability, and acceptability. Trust in the chatbot, or in the chatbot provider, was commonly reported as being an important factor. Studies that reported on user acceptability generally found that the chatbots were rated highly, regardless of its type. The number of people seeking health information via digital methods throughout the pandemic has increased [3]; this may have played a role in this acceptability. Digital ability or perceived IT skills were not reported to have a large effect on chatbot use. In some studies, women used the chatbots more than men, and although there were differences in chatbot use or perspectives reported by age, it was clear that chatbots are being used and enjoyed by older people as well as those who are younger. There was no clear indication to whether age and gender play a role in openness to using these tools.

Research presenting results from the United States was overrepresented (5/9, 56% of studies), and the results seen here indicate a need for more diverse reporting and evaluation with COVID-19 chatbots. There have been broader calls in the literature for chatbots responding to COVID-19 to take a global view and be developed with researchers and local data scientists from low- and middle-income countries, to enable better access on a global level [32].

The COVID-19 pandemic has presented challenges to health services and authorities globally, with solutions being developed and rolled out to support countries with accurate and timely health information at a rapid speed. This expediated time frame may have resulted in some of the usual design processes, such as user consultations, message testing, and literature reviews, having not been conducted prior to launch, resulting in phases of evaluation and iteration in real-world settings. The challenge of managing the amount of content in a fast-moving landscape was described by one national government chatbot [30], and partnerships with technical teams were reported as being beneficial [25]. New processes were developed and chatbots have helped to quickly disseminate high-quality information in a time of crisis. These processes and learnings from implementation will be key in informing chatbots for future health emergencies. As well as information delivery, chatbots have the potential to inform information provision and pandemic response by analyzing the questions people are asking or looking at the information they are searching for to identify information voids.

The review revealed a lack of standardized reporting on user experience and user preferences. Although this may be partially explained by the need for expedited deployment, it will be important in the future to revisit and reaffirm best practices in user-focused design and acceptance testing to ensure that these tools are effective. Developing standardized measures for reporting on user experiences with chatbots will help to synthesize the evidence and move it forward in a more cohesive manner. The literature suggests that working in a co-design model involving end users can help with engagement [7], and that usability, enjoyment, and trust are also key factors [9].

It is apparent that further research is needed to better understand user experience, engagement, and uptake, particularly more research from countries outside of the United States. The COVID-19 pandemic has had an unprecedented effect on digital health [33]. Lessons learned on how best to reach people with health information and interventions during the COVID-19 pandemic will have broad application to other health emergencies and future pandemics.

The scoping review summarized the evidence about user experience with COVID-19 chatbots, but as the goal of a scoping review is to map the literature, study quality was not assessed and data synthesis did not occur. The studies that were included reported diverse aspects of user experience, resulting in limitations in drawing together findings on particular components.

The COVID-19 pandemic presented a unique and specific challenge for digital health interventions in that design and implementation were required at a rapid speed as digital health service adoption was accelerated across the globe. This paper adds to the literature by describing what is known about this rapid implementation process in terms of user experience and user uptake and provides guidance for future tools, as well as directions for future research. This paper has shown both the potential of chatbots to reach users in an emergency and the need to better understand how users engage and what they want.