This paper is in the following e-collection/theme issue:

Chatbots and Conversational Agents (542) Artificial Intelligence (1409) Reviews in Medical Informatics (93)

Published on in Vol 22, No 6 (2020): June

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/18301, first published .
Technical Metrics Used to Evaluate Health Care Chatbots: Scoping Review

Technical Metrics Used to Evaluate Health Care Chatbots: Scoping Review

Technical Metrics Used to Evaluate Health Care Chatbots: Scoping Review

Authors of this article:

Alaa Abd-Alrazaq1 Author Orcid Image ;   Zeineb Safi1 Author Orcid Image ;   Mohannad Alajlani2 Author Orcid Image ;   Jim Warren3 Author Orcid Image ;   Mowafa Househ1 Author Orcid Image ;   Kerstin Denecke4 Author Orcid Image
Article Authors Cited by (73) Tweetations (12) Metrics

Journals

  1. Zhang J, Oh Y, Lange P, Yu Z, Fukuoka Y. Artificial Intelligence Chatbot Behavior Change Model for Designing Artificial Intelligence Chatbots to Promote Physical Activity and a Healthy Diet: Viewpoint. Journal of Medical Internet Research 2020;22(9):e22845 View
  2. Omoregbe N, Ndaman I, Misra S, Abayomi-Alli O, Damaševičius R, Dogra A. Text Messaging-Based Medical Diagnosis Using Natural Language Processing and Fuzzy Logic. Journal of Healthcare Engineering 2020;2020:1 View
  3. Lin J, Joseph T, Parga-Belinkie J, Mandel A, Schumacher R, Neumann K, Scalise L, Gaulton J, Christ L, Leitner K, Rosin R. Development of a practical training method for a healthcare artificial intelligence (AI) chatbot. BMJ Innovations 2021;7(2):441 View
  4. Issom D, Hardy-Dessources M, Romana M, Hartvigsen G, Lovis C. Toward a Conversational Agent to Support the Self-Management of Adults and Young Adults With Sickle Cell Disease: Usability and Usefulness Study. Frontiers in Digital Health 2021;3 View
  5. Abd-Alrazaq A, Alajlani M, Ali N, Denecke K, Bewick B, Househ M. Perceptions and Opinions of Patients About Mental Health Chatbots: Scoping Review. Journal of Medical Internet Research 2021;23(1):e17828 View
  6. Bérubé C, Schachner T, Keller R, Fleisch E, v Wangenheim F, Barata F, Kowatsch T. Voice-Based Conversational Agents for the Prevention and Management of Chronic and Mental Health Conditions: Systematic Literature Review. Journal of Medical Internet Research 2021;23(3):e25933 View
  7. Wang J, Hwang G, Chang C. Directions of the 100 most cited chatbot-related human behavior research: A review of academic publications. Computers and Education: Artificial Intelligence 2021;2:100023 View
  8. Kolenik T, Gams M. Intelligent Cognitive Assistants for Attitude and Behavior Change Support in Mental Health: State-of-the-Art Technical Review. Electronics 2021;10(11):1250 View
  9. Mokmin N, Ibrahim N. The evaluation of chatbot as a tool for health literacy education among undergraduate students. Education and Information Technologies 2021;26(5):6033 View
  10. Hwang G, Chang C. A review of opportunities and challenges of chatbots in education. Interactive Learning Environments 2023;31(7):4099 View
  11. Figueroa C, Luo T, Jacobo A, Munoz A, Manuel M, Chan D, Canny J, Aguilera A. Conversational Physical Activity Coaches for Spanish and English Speaking Women: A User Design Study. Frontiers in Digital Health 2021;3 View
  12. Oh Y, Zhang J, Fang M, Fukuoka Y. A systematic review of artificial intelligence chatbots for promoting physical activity, healthy diet, and weight loss. International Journal of Behavioral Nutrition and Physical Activity 2021;18(1) View
  13. Denecke K, Schmid N, Nüssli S. Implementation of Cognitive Behavioral Therapy in e–Mental Health Apps: Literature Review. Journal of Medical Internet Research 2022;24(3):e27791 View
  14. Chagas B, Pagano A, Prates R, Praes E, Ferreguetti K, Vaz H, Reis Z, Ribeiro L, Ribeiro A, Pedroso T, Beleigoli A, Oliveira C, Marcolino M. Evaluating User Experience With a Chatbot Designed as a Public Health Response to the COVID-19 Pandemic in Brazil: Mixed Methods Study. JMIR Human Factors 2023;10:e43135 View
  15. Barletta V, Caivano D, Colizzi L, Dimauro G, Piattini M. Clinical-chatbot AHP evaluation based on “quality in use” of ISO/IEC 25010. International Journal of Medical Informatics 2023;170:104951 View
  16. Ren R, Zapata M, Castro J, Dieste O, Acuna S. Experimentation for Chatbot Usability Evaluation: A Secondary Study. IEEE Access 2022;10:12430 View
  17. Denecke K, May R. Developing a Technical-Oriented Taxonomy to Define Archetypes of Conversational Agents in Health Care: Literature Review and Cluster Analysis. Journal of Medical Internet Research 2023;25:e41583 View
  18. Parmar P, Ryu J, Pandya S, Sedoc J, Agarwal S. Health-focused conversational agents in person-centered care: a review of apps. npj Digital Medicine 2022;5(1) View
  19. Luo B, Lau R, Li C, Si Y. A critical review of state‐of‐the‐art chatbot designs and applications. WIREs Data Mining and Knowledge Discovery 2022;12(1) View
  20. Mauriello M, Tantivasadakarn N, Mora-Mendoza M, Lincoln E, Hon G, Nowruzi P, Simon D, Hansen L, Goenawan N, Kim J, Gowda N, Jurafsky D, Paredes P. A Suite of Mobile Conversational Agents for Daily Stress Management (Popbots): Mixed Methods Exploratory Study. JMIR Formative Research 2021;5(9):e25294 View
  21. Ta-Johnson V, Boatfield C, Wang X, DeCero E, Krupica I, Rasof S, Motzer A, Pedryc W. Assessing the Topics and Motivating Factors Behind Human-Social Chatbot Interactions: Thematic Analysis of User Experiences. JMIR Human Factors 2022;9(4):e38876 View
  22. Lahti L. Detecting the patient’s need for help with machine learning based on expressions. BMC Medical Research Methodology 2022;22(1) View
  23. Parviainen J, Rantala J. Chatbot breakthrough in the 2020s? An ethical reflection on the trend of automated consultations in health care. Medicine, Health Care and Philosophy 2022;25(1):61 View
  24. Sagstad M, Morken N, Lund A, Dingsør L, Nilsen A, Sorbye L. Quantitative User Data From a Chatbot Developed for Women With Gestational Diabetes Mellitus: Observational Study. JMIR Formative Research 2022;6(4):e28091 View
  25. White B, Martin A, White J. User Experience of COVID-19 Chatbots: Scoping Review. Journal of Medical Internet Research 2022;24(12):e35903 View
  26. Denecke K, Abd-Alrazaq A, Househ M, Warren J. Evaluation Metrics for Health Chatbots: A Delphi Study. Methods of Information in Medicine 2021;60(05/06):171 View
  27. Ali H, Niazi I, Russell B, Crofts C, Madanian S, White D. Review of Time Domain Electronic Medical Record Taxonomies in the Application of Machine Learning. Electronics 2023;12(3):554 View
  28. Noble J, Zamani A, Gharaat M, Merrick D, Maeda N, Lambe Foster A, Nikolaidis I, Goud R, Stroulia E, Agyapong V, Greenshaw A, Lambert S, Gallson D, Porter K, Turner D, Zaiane O. Developing, Implementing, and Evaluating an Artificial Intelligence–Guided Mental Health Resource Navigation Chatbot for Health Care Workers and Their Families During and Following the COVID-19 Pandemic: Protocol for a Cross-sectional Study. JMIR Research Protocols 2022;11(7):e33717 View
  29. Hayakawa M, Watanabe O, Shiga K, Fujishita M, Yamaki C, Ogo Y, Takahashi T, Ikeguchi Y, Takayama T. Exploring types of conversational agents for resolving cancer patients’ questions and concerns: Analysis of 100 telephone consultations on breast cancer. Patient Education and Counseling 2023;106:75 View
  30. Chan W, Fitzsimmons-Craft E, Smith A, Firebaugh M, Fowler L, DePietro B, Topooco N, Wilfley D, Taylor C, Jacobson N. The Challenges in Designing a Prevention Chatbot for Eating Disorders: Observational Study. JMIR Formative Research 2022;6(1):e28003 View
  31. Kaushik A, Jacob B, Velavan P. An Exploratory Study on a Reinforcement Learning Prototype for Multimodal Image Retrieval Using a Conversational Search Interface. Knowledge 2022;2(1):116 View
  32. He Y, Yang L, Zhu X, Wu B, Zhang S, Qian C, Tian T. Mental Health Chatbot for Young Adults With Depressive Symptoms During the COVID-19 Pandemic: Single-Blind, Three-Arm Randomized Controlled Trial. Journal of Medical Internet Research 2022;24(11):e40719 View
  33. Jabir A, Martinengo L, Lin X, Torous J, Subramaniam M, Tudor Car L. Evaluating Conversational Agents for Mental Health: Scoping Review of Outcomes and Outcome Measurement Instruments. Journal of Medical Internet Research 2023;25:e44548 View
  34. Denecke K. Framework for Guiding the Development of High-Quality Conversational Agents in Healthcare. Healthcare 2023;11(8):1061 View
  35. Liu C, Zowghi D, Peng G, Kong S. Information quality of conversational agents in healthcare. Information Development 2023 View
  36. Giansanti D. The Chatbots Are Invading Us: A Map Point on the Evolution, Applications, Opportunities, and Emerging Problems in the Health Domain. Life 2023;13(5):1130 View
  37. Simhadri Apparao Polireddi N, Kavitha J. Effectiveness of automated chatbots for operational support and self-service based on fuzzy classifier for ASD. Soft Computing 2023 View
  38. Talyshinskii A, Naik N, Hameed B, Juliebø-Jones P, Somani B. Potential of AI-Driven Chatbots in Urology: Revolutionizing Patient Care Through Artificial Intelligence. Current Urology Reports 2024;25(1):9 View
  39. Wang X, Luo R, Liu Y, Chen P, Tao Y, He Y. Revealing the complexity of users’ intention to adopt healthcare chatbots: A mixed-method analysis of antecedent condition configurations. Information Processing & Management 2023;60(5):103444 View
  40. Andrews N, Ireland D, Vijayakumar P, Burvill L, Hay E, Westerman D, Rose T, Schlumpf M, Strong J, Claus A. Acceptability of a Pain History Assessment and Education Chatbot (Dolores) Across Age Groups in Populations With Chronic Pain: Development and Pilot Testing. JMIR Formative Research 2023;7:e47267 View
  41. López-Sánchez J, Bermeo-Giraldo M, Valencia-Arias A, Bao García R, Gallegos A. Chatbots and young people in emerging economies: Factors affecting user satisfaction. Cogent Social Sciences 2023;9(2) View
  42. Griffin A, Khairat S, Bailey S, Chung A. A chatbot for hypertension self-management support: user-centered design, development, and usability testing. JAMIA Open 2023;6(3) View
  43. Kiuchi K, Otsu K, Hayashi Y. Psychological insights into the research and practice of embodied conversational agents, chatbots and social assistive robots: a systematic meta-review. Behaviour & Information Technology 2024;43(15):3696 View
  44. Ding H, Simmich J, Vaezipour A, Andrews N, Russell T. Evaluation framework for conversational agents with artificial intelligence in health interventions: a systematic scoping review. Journal of the American Medical Informatics Association 2024;31(3):746 View
  45. Jiang Z, Huang X, Wang Z, Liu Y, Huang L, Luo X. Embodied Conversational Agents for Chronic Diseases: Scoping Review. Journal of Medical Internet Research 2024;26:e47134 View
  46. Gengatharan D, Saggi S, Bin Abd Razak H. Pre-operative Planning of High Tibial Osteotomy With ChatGPT: Are We There Yet?. Cureus 2024 View
  47. Chou Y, Lin C, Lee S, Lee Y, Cheng L. User-Friendly Chatbot to Mitigate the Psychological Stress of Older Adults During the COVID-19 Pandemic: Development and Usability Study. JMIR Formative Research 2024;8:e49462 View
  48. Laymouna M, Ma Y, Lessard D, Schuster T, Engler K, Lebouché B. Roles, Users, Benefits, and Limitations of Chatbots in Health Care: Rapid Review. Journal of Medical Internet Research 2024;26:e56930 View
  49. Nadarzynski T, Knights N, Husbands D, Graham C, Llewellyn C, Buchanan T, Montgomery I, Ridge D, Silva J. Achieving health equity through conversational AI: A roadmap for design and implementation of inclusive chatbots in healthcare. PLOS Digital Health 2024;3(5):e0000492 View
  50. Sezgin E, Kocaballi A, Dolce M, Skeens M, Militello L, Huang Y, Stevens J, Kemper A. Chatbot for Social Need Screening and Resource Sharing With Vulnerable Families: Iterative Design and Evaluation Study. JMIR Human Factors 2024;11:e57114 View
  51. Martins L, Fernández-Ferrer M, Puertas E. Analysing Conversation Pathways with a Chatbot Tutor to Enhance Self-Regulation in Higher Education. Education Sciences 2024;14(6):590 View
  52. Izadi S, Forouzanfar M. Error Correction and Adaptation in Conversational AI: A Review of Techniques and Applications in Chatbots. AI 2024;5(2):803 View
  53. Viswanathan V, Jain V, Vaish A, Jeyaraman M, Iyengar K, Vaishya R. Chatbots and Their Applications in Medical Fields: Current Status and Future Trends: A Scoping Review. Apollo Medicine 2024;21(4):386 View
  54. Wahab N, Forsyth R. Experiences of patients with hard-to-heal wounds: insights from a pilot survey. Journal of Wound Care 2024;33(10):788 View
  55. Wang H, Masselos K, Tong J, Connor H, Scully J, Zhang S, Rafla D, Posarelli M, Tan J, Agar A, Kalloniatis M, Phu J. ChatGPT for Addressing Patient-centered Frequently Asked Questions in Glaucoma Clinical Practice. Ophthalmology Glaucoma 2024 View
  56. Indah Permatasari R, Parama Artha D, Satria Wiratama B, Wulandari H, Sato H, El Wahed A, Frutos R, Wibawa T, Prabandari R, Hasanbasri M, Ramadona A, Rauf A, Wiratama B, Maula A. Overview of chatbot usage on mental health: A scoping review. BIO Web of Conferences 2024;132:05002 View
  57. Agarwal S. Developing An Artificial Intelligence Based Chatbot For And Early And Effective Prediction Of Various Diseases. International Journal of Research in Medical Sciences and Technology 2024;17(1):76 View
  58. Su B, Jones R, Chen K, Kostenko E, Schmid M, DeMaria A, Villa A, Swarup M, Weida J, Tuuli M. Chatbot for patient education for prenatal aneuploidy testing: A multicenter randomized controlled trial. Patient Education and Counseling 2025;131:108557 View

Books/Policy Documents

  1. Gamberini L, Pluchino P, Porto L, Zordan F, Monterosso A, Palanivel K, Fontanari A. HCI International 2021 - Posters. View
  2. Liu C, Zhang B, Peng G. Distributed, Ambient and Pervasive Interactions. View
  3. Gibbons M, Shaihk Y, Ayalasomayajula F. Healthcare Information Management Systems. View
  4. Alamoudi Y, Alasmari H, Alamoudi G, Alghamdi H. Intelligent Systems and Applications. View