Published on in Vol 26 (2024)

Preprints (earlier versions) of this paper are available at, first published .
Chatbots and COVID-19: Taking Stock of the Lessons Learned

Chatbots and COVID-19: Taking Stock of the Lessons Learned

Chatbots and COVID-19: Taking Stock of the Lessons Learned


1Department of Social Determinants of Health, Division of UHC Healthier Populations, World Health Organization, Geneva, Switzerland

2Department of Epidemic and Pandemic Preparedness and Prevention, Emergencies Programme, World Health Organization, Geneva, Switzerland

3University of Memphis School of Public Health, Memphis, TN, United States

4Alliance for Health Policy and Systems Research, Science Division, World Health Organization, Geneva, Switzerland

5Department of Digital Health and Innovation, Science Division, World Health Organization, Geneva, Switzerland

6Department of Health Promotion, Division of UHC Healthier Populations, World Health Organization, Geneva, Switzerland

Corresponding Author:

Hebe Gouda, MPH, PhD

Department of Health Promotion

Division of UHC Healthier Populations

World Health Organization

Avenue Appia 20

Geneva, 1211


Phone: 41 793865070


While digital innovation in health was already rapidly evolving, the COVID-19 pandemic has accelerated the generation of digital technology tools, such as chatbots, to help increase access to crucial health information and services to those who were cut off or had limited contact with health services. This theme issue titled “Chatbots and COVID-19” presents articles from researchers and practitioners across the globe, describing the development, implementation, and evaluation of chatbots designed to address a wide range of health concerns and services. In this editorial, we present some of the key challenges and lessons learned arising from the content of this theme issue. Most notably, we note that a stronger evidence base is needed to ensure that chatbots and other digital tools are developed to best serve the needs of population health.

J Med Internet Res 2024;26:e54840



Like many areas of technology, digital innovation in health is evolving rapidly [1]. It is frequently used to expand access to health services for populations whose options are otherwise restricted due to geographical, economic, or sociocultural barriers. Digital technology has been increasingly used in health care and public health to help disseminate important health-related information to the public as well as health care workers, and to support health services by scheduling medical appointments, for example, or even helping practitioners with the diagnosis of health problems and monitoring health outcomes. During 2020 to 2023, the COVID-19 pandemic led to increased use of digital technology as countries limited in-person access to health care facilities, paused or slowed immunization and other health campaigns, and reallocated health resources to the COVID-19 response. Suddenly, access to health care and essential health services was drastically reduced at a time when health care needs and concerns were increasing.

As a result, there was a tremendous surge of interest in digital technology, including chatbots—computer programs that present a conversation-like interface through which people can access information and services [2]. During the early stages of the COVID-19 pandemic, national and subnational governments, as well as international health organizations such as the World Health Organization (WHO) [3,4] and the United States Centers for Disease Control and Prevention (CDC) [5], private companies, nongovernmental organizations, and researchers explored these new channels to reach people who were otherwise cut off from health services.

Chatbots have been designed to provide a range of health-related services, from information sharing [6] to offering support for mental well-being [7,8], and diagnostic tools to assist health care workers [9]. They have been designed for a range of subgroups such as specific age-groups (eg, youth who are often considered the most likely group to accept and adopt mobile technology), people with specific diseases (eg, young people with diabetes [10]), or those in health care roles (eg, frontline health workers [11]).

While this theme issue in the Journal of Medical Internet Research titled “Chatbots and COVID-19” [12] focuses on chatbots related to COVID-19, the span of topics covered by these chatbots is broad. They range from vaccine hesitancy and equity in access to relevant information [13-15] to pandemic-related mental health [16-19]. A number of topics focus on health promotion interventions, such as smoking cessation [20], as well as encouraging physical activity [21] and tooth-brushing [22]. Although the topics are diverse, some common themes have emerged from these explorations, namely, the importance of considering user experience in chatbot design, the need to make chatbots inclusive, and the importance of user trust [23].

The perception of chatbots in these studies was generally positive: user experiences demonstrate that the chatbots are seen as valuable tools for managing personal health [24-26] and are considered useful in health care and clinical settings [27]. However, there was a clear paucity of robust evidence on the impact these chatbots have on health outcomes. Chatbots are a medium for delivering content, but the quality, relevance, and impact of the content itself is as essential. Not all chatbots are equal, and potential efficiency gains should not be assumed to deliver automatically. Some chatbots may potentially do more harm than good. There is an urgent need for more implementation research on best-practice monitoring and evaluation of chatbots in terms of their contribution to health, as well as their strengths and weaknesses compared to other mediums when used in different contexts or with different populations.

It is also important to not lose sight of the audience. In some cases, focusing on a chatbot at the expense of alternative approaches may lead to the marginalization of certain population groups. There are inherent access barriers created by chatbots. Most notably, of course, is the access to the mobile technology itself, which is not equally distributed globally. Although mobile phones have become ubiquitous globally, mobile data can be unreliable and expensive in many contexts and the exclusion of certain groups due to age or language restrictions, for example, can be a major challenge [28]. Nonetheless, chatbots can also be a valuable tool for increasing access by overcoming literacy barriers for some populations and some globally focused chatbots have already been developed in multiple languages. Pahola (a chatbot designed to help people reduce their alcohol consumption [3]) and Florence (a chatbot designed to help people with multitude of health concerns, including smoking cessation [4]), for example, “speak” 4 and 7 different languages, respectively.

There is reason to be optimistic about chatbots and new digital tools for health. They hold the potential to remove access barriers and engage marginalized populations. Over the past 2 decades, there has been some hype around these innovations, however, and excessive confidence about the potential benefits of chatbots despite a lack of evidence on their quantifiable impact on health outcomes. Furthermore, as recent debates on the role of artificial intelligence (AI) showcase, technology can trigger unintended consequences to both its users and wider society. Although chatbots are not all the same as AI—indeed, many in this series are often closer to traditional SMS text messaging than to a machine learning model—a similar awareness of context and opportunity costs is essential for their healthy development. To fully understand the benefits of health chatbots, providers must work closely with a broad spectrum of perspectives, including local communities, social scientists, anthropologists, and ethicists, to ensure that their development addresses growing concerns about digital harms [2,29]. Without adequate ethical guidance, transparency, and accountability of developers and products, there is a risk of harming users through intentional or unintentional discrimination, marginalization, or lack of privacy [29].

In May 2023, WHO officially declared the end of the COVID-19 international health emergency. Nevertheless, its consequences for health systems and digital health development will continue to reverberate for some time. A wider interest in health chatbots may have started as a stopgap, but their use and value have not ended with the pandemic. However, chatbots must adhere to the same evidence-based standards as any new tool in a public health or clinical setting. Without this robust foundation, chatbots will remain tools that are at best a nice-to-have, and at worst only considered when other options are unavailable.

Authors' Contributions

VA conceived the project. VA and HG drafted the editorial. All authors reviewed and edited the manuscript.

Conflicts of Interest

All authors acted as editors of the special theme issue “Chatbots and COVID-19” in the Journal of Medical Internet Research at the time of this publication.

  1. Luengo-Oroz M, Hoffmann Pham K, Bullock J, Kirkpatrick R, Luccioni A, Rubel S, et al. Artificial intelligence cooperation to support the global response to COVID-19. Nat Mach Intell. May 22, 2020;2(6):295-297. [CrossRef]
  2. Wilson L, Marasoiu M. The development and use of chatbots in public health: scoping review. JMIR Hum Factors. Oct 05, 2022;9(4):e35882. [FREE Full text] [CrossRef] [Medline]
  3. Monteiro MG, Pantani D, Pinsky I, Hernandes Rocha TA. Using the Pan American Health Organization digital conversational agent to educate the public on alcohol use and health: preliminary analysis. JMIR Form Res. Apr 06, 2023;7:e43165. [FREE Full text] [CrossRef] [Medline]
  4. Loveys K, Lloyd E, Sagar M, Broadbent E. Development of a virtual human for supporting tobacco cessation during the COVID-19 pandemic. J Med Internet Res. Dec 05, 2023;25:e42310. [FREE Full text] [CrossRef] [Medline]
  5. Shah A, Oyegun E, Hampton W, Neri A, Maddox N, Raso D, et al. Engagement with the Centers for Disease Control and Prevention coronavirus self-checker and guidance provided to users in the United States from March 23, 2020, to April 19, 2021: thematic and trend analysis. J Med Internet Res. Mar 10, 2023;25:e39054. [FREE Full text] [CrossRef] [Medline]
  6. Nehme M, Schneider F, Perrin A, Sum Yu W, Schmitt S, Violot G, et al. The development of a chatbot technology to disseminate post-COVID-19 information: descriptive implementation study. J Med Internet Res. Jun 05, 2023;25:e43113. [FREE Full text] [CrossRef] [Medline]
  7. Islam A, Chaudhry BM. Design validation of a relational agent by COVID-19 patients: mixed methods study. JMIR Hum Factors. Jun 08, 2023;10:e42740. [FREE Full text] [CrossRef] [Medline]
  8. Wang H, Lin T, Yao Y, Hsu C, Yang C, Chen TH, et al. Community-based digital contact tracing of emerging infectious diseases: design and implementation study with empirical COVID-19 cases. J Med Internet Res. Nov 08, 2023;25:e47219. [FREE Full text] [CrossRef] [Medline]
  9. Han J, Montagna M, Grammenos A, Xia T, Bondareva E, Siegele-Brown C, et al. Evaluating listening performance for COVID-19 detection by clinicians and machine learning: comparative study. J Med Internet Res. May 9, 2023;25:e44804. [CrossRef]
  10. Boggiss A, Consedine N, Hopkins S, Silvester C, Jefferies C, Hofman P, et al. Improving the well-being of adolescents with Type 1 Diabetes during the COVID-19 pandemic: qualitative study exploring acceptability and clinical usability of a self-compassion chatbot. JMIR Diabetes. May 05, 2023;8:e40641. [FREE Full text] [CrossRef] [Medline]
  11. Jackson-Triche M, Vetal D, Turner E, Dahiya P, Mangurian C. Meeting the behavioral health needs of health care workers during COVID-19 by leveraging chatbot technology: development and usability study. J Med Internet Res. Jun 08, 2023;25:e40635. [FREE Full text] [CrossRef] [Medline]
  12. Theme Issue 2022: Chatbots and COVID-19. Journal of Medical Internet Research. JMIR Publications URL: [accessed 2023-12-28]
  13. Perez-Ramos JG, Leon-Thomas M, Smith SL, Silverman L, Perez-Torres C, Hall WC, et al. COVID-19 vaccine equity and access: case study for health care chatbots. JMIR Form Res. Jan 25, 2023;7:e39045. [FREE Full text] [CrossRef] [Medline]
  14. Trzebiński W, Claessens T, Buhmann J, De Waele A, Hendrickx G, Van Damme P, et al. The effects of expressing empathy/autonomy support using a COVID-19 vaccination chatbot: experimental study in a sample of Belgian adults. JMIR Form Res. May 08, 2023;7:e41148. [FREE Full text] [CrossRef] [Medline]
  15. Luk TT, Lui JHT, Wang MP. Efficacy, usability, and acceptability of a chatbot for promoting COVID-19 vaccination in unvaccinated or booster-hesitant young adults: pre-post pilot study. J Med Internet Res. Oct 04, 2022;24(10):e39063. [FREE Full text] [CrossRef] [Medline]
  16. Ludin N, Holt-Quick C, Hopkins S, Stasiak K, Hetrick S, Warren J, et al. A chatbot to support young people during the COVID-19 pandemic in New Zealand: evaluation of the real-world rollout of an open trial. J Med Internet Res. Nov 04, 2022;24(11):e38743. [FREE Full text] [CrossRef] [Medline]
  17. He Y, Yang L, Zhu X, Wu B, Zhang S, Qian C, et al. Mental health chatbot for young adults with depressive symptoms during the COVID-19 pandemic: single-Blind, three-arm randomized controlled trial. J Med Internet Res. Nov 21, 2022;24(11):e40719. [FREE Full text] [CrossRef] [Medline]
  18. Goonesekera Y, Donkin L. A cognitive behavioral therapy chatbot (Otis) for health anxiety management: mixed methods pilot study. JMIR Form Res. Oct 20, 2022;6(10):e37877. [FREE Full text] [CrossRef] [Medline]
  19. Sinha C, Meheli S, Kadaba M. Understanding digital mental health needs and usage with an artificial intelligence-led mental health app (Wysa) during the COVID-19 pandemic: retrospective analysis. JMIR Form Res. Jan 26, 2023;7:e41913. [FREE Full text] [CrossRef] [Medline]
  20. Whittaker R, Dobson R, Garner K. Chatbots for smoking cessation: scoping review. J Med Internet Res. Sep 26, 2022;24(9):e35556. [FREE Full text] [CrossRef] [Medline]
  21. Wlasak W, Zwanenburg SP, Paton C. Supporting autonomous motivation for physical activity with chatbots during the COVID-19 pandemic: factorial experiment. JMIR Form Res. Jan 25, 2023;7:e38500. [FREE Full text] [CrossRef] [Medline]
  22. Pithpornchaiyakul S, Naorungroj S, Pupong K, Hunsrisakhun J. Using a chatbot as an alternative approach for in-person toothbrushing training during the COVID-19 pandemic: comparative study. J Med Internet Res. Oct 21, 2022;24(10):e39218. [FREE Full text] [CrossRef] [Medline]
  23. Weeks R, Sangha P, Cooper L, Sedoc J, White S, Gretz S, et al. Usability and credibility of a COVID-19 vaccine chatbot for Yyoung adults and health workers in the United States: formative mixed methods study. JMIR Hum Factors. Jan 30, 2023;10:e40533. [FREE Full text] [CrossRef] [Medline]
  24. Chagas BA, Pagano AS, Prates RO, Praes EC, Ferreguetti K, Vaz H, et al. Evaluating user experience with a chatbot designed as a public health response to the COVID-19 pandemic in Brazil: mixed methods study. JMIR Hum Factors. Apr 03, 2023;10:e43135. [FREE Full text] [CrossRef] [Medline]
  25. White BK, Martin A, White JA. User experience of COVID-19 chatbots: scoping review. J Med Internet Res. Dec 27, 2022;24(12):e35903. [FREE Full text] [CrossRef] [Medline]
  26. Shan Y, Ji M, Xie W, Zhang X, Qian X, Li R, et al. Use of health care chatbots among young people in China during the Omicron wave of COVID-19: evaluation of the user experience of and satisfaction with the technology. JMIR Hum Factors. Jun 09, 2022;9(2):e36831. [FREE Full text] [CrossRef] [Medline]
  27. Daniel T, de Chevigny A, Champrigaud A, Valette J, Sitbon M, Jardin M, et al. Answering hospital caregivers' questions at any time: proof-of-concept study of an artificial intelligence-based chatbot in a French hospital. JMIR Hum Factors. Oct 11, 2022;9(4):e39102. [FREE Full text] [CrossRef] [Medline]
  28. Phiri M, Munoriyarwa A. Health chatbots in Africa: scoping review. J Med Internet Res. Jun 14, 2023;25:e35573. [FREE Full text] [CrossRef] [Medline]
  29. Fournier-Tombs E, McHardy J. A medical ethics framework for conversational artificial intelligence. J Med Internet Res. Jul 26, 2023;25:e43068. [FREE Full text] [CrossRef] [Medline]

AI: artificial intelligence
CDC: Centers for Disease Control and Prevention
WHO: World Health Organization

Edited by T de Azevedo Cardoso; This is a non–peer-reviewed article. submitted 23.11.23; accepted 20.12.23; published 21.03.24.


©Virginia Arnold, Tina D Purnat, Robert Marten, Andrew Pattison, Hebe Gouda. Originally published in the Journal of Medical Internet Research (, 21.03.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.