Health care CAs often operate within particular domains of health care that require the integration of domain-specific information and language. This theme is concerned with the challenges of obtaining the required problem domain information, collecting data to train CAs, and integrating CAs with the existing systems and infrastructures. For example, a CA to be designed for helping general practitioners in the primary care domain needs to be trained by a large number of doctor-patient conversations. Obtaining this kind of information is challenging and resource intensive. In addition, the expert medical knowledge needs to be translated into conversational format. Similarly, CAs cannot be developed as isolated applications: they need to be integrated with the existing systems and infrastructure.

Two projects (P6 and P14) reported on the difficulties of obtaining domain information, in these cases, for health care enterprise and medical adherence communications. These difficulties are associated with the lack of time of domain experts (P6), the domain information being not in a conversational format (P6) or being distributed across many subdomains (P6), and the knowledge acquisition bottleneck (P14). While P6 recruited subject matter experts to review conversation flows and response frames to format health care data for conversation-like interactions, P14 implemented incremental acquisition from the expert or user to alleviate the problem of obtaining domain-specific information.

Three studies reported some challenges associated with integrating CAs into existing infrastructures. One project (P15) using CAs to support health care tracking of children with special needs at home explained the security and privacy challenges of integrating CA-collected information into clinical systems and workflows. Using interoperability standards (eg, Fast Healthcare Interoperability Resources) to share caregiver notes with health care professionals was suggested as one of the fundamental approaches for integration (P15). Many CAs are developed using underlying development platforms such as Amazon Alexa Skills or Google Dialogflow. P14 highlighted the difficulty in adapting to the rapidly evolving platforms that can potentially make the CAs designed according to the previous platform functionality stop working. Finally, P11 discussed the tension between the ubiquity and richness of the underlying platforms. For example, while SMS-based text messaging is ubiquitous, it is purely text based and does not offer some of the multimodal interaction options of other messaging platforms such as Facebook Messenger or WhatsApp (eg, 1-click suggested response buttons or carousel menus). P11 used SMS text message–based messaging to support a higher degree of accessibility and ubiquity; however, to expand the richness of the user experience they included more immersive education content with multimedia messages including infographics to elaborate on each health goal option.

This was one of the major themes that captured several challenges related to the characteristics and qualities of interaction between users and CAs, and the ways in which users and CAs work together. The subthemes included Personalization of CAs, Relationship Building Between CAs and Users, Safety and Privacy, and User Engagement. Some common challenges were supporting users’ trust in CAs, enabling CAs to show empathy, and ensuring users’ privacy.

Personalization emerged as an essential design feature as part of the projects. Personalization was reported as a challenge in 10 projects. The personalization challenges were providing appropriate responses based on users’ context (P8 and P3), tailoring conversations for different user groups with a broad age range and different health literacy levels (P14, P6, P9, P11, and P12), and minimizing the question overload in surveys (P13). While 1 project reported on the difficulties in evaluating the effects on adaptive features (P6), another project presented the challenges associated with designing for dyads (P5), which include a patient and a caregiver. P5 explained that dyads using the same device pose several challenges, including dyads’ difficulties managing technology, dyads’ overall wide range of technology literacy, and how to support both members of the dyad through the functionality of the CA.

Humans and CAs may have different forms of relationships ranging from very short-term relationships typically characterized by one-off task-based conversational exchanges to long-term relationships in which CAs have longer conversational interactions across different topics over a longer period. Relationship building—how a human-CA relationship is established and maintained—was reported as a challenge in 13 projects. Trust (P3, P4, P7, P13, and P16), empathy (P4, P17, and P16), self-disclosure (P7), and transparency (P12) were presented as important dimensions of relationship building. One project focusing on designing a CA for people with visual impairment (P4) explains the importance of CAs to show empathy in establishing and strengthening trust between the sighted Aira agent and the client with visual impairment. According to P4, to incorporate empathy into conversational interaction, an agent must represent not only the situation itself (eg, what objects are present, their spatial relationships, movement vectors) but also the other agent’s experience and interpretation of the situation. A project designing a CA for older adults’ care (P3) describes the difficulties of building trust due to concerns about security, legal issues, the sharing and storing of personal and sensitive information, and privacy and ethical concerns, among others [59]. Issues with any of these and other concerns have the potential to damage the relationship critically or terminally between the CA and the patient. To address this challenge, P3 incorporated trust-building and repair strategies from the onset. Previous research has shown that reliability is likely the preeminent factor in building trust between the patient and the artificial care agent [60,61]. By contrast, other factors such as competence, benevolence, and integrity are also likely to significantly influence [62]. Transferring users’ self-disclosure was reported as a challenge in human-in-the-loop artificial intelligence systems where CAs mediate between a user and a domain expert (P7).

Safety of user data and privacy of personal information shared with CAs have influenced the user decision and perception toward CAs. Six projects reported users’ safety (P3 and P13) and privacy (P5, P7, P8, and P10) as challenges. P10 explained that while some users were worried that using the voice assistant would disturb others, other users felt discomfort or concerned with privacy when other people could hear them track their food. To tackle this issue, some suggested solutions in P10 included adopting other devices or modalities for input (eg, taking picture or text input using phone or web), as well as implementing food “template” features for quick commands (eg, saying “Alexa, journal number 1” or “Google, journal same breakfast as yesterday”). In the cases of CAs as mediators between 2 users (P7) or multiple users (P5), privacy challenges may become more critical. As many health care CAs deal with safety-critical user information and decision making, P3 and P13 noted the need for more strict and standardized evaluation measures for CAs.

Establishing user engagement strategies for CAs is fundamental to improving user experience and sustained use. However, 6 projects reported enabling and maintaining user engagement as a challenge (P4, P7, P9, P10, P13, and P14). P9 explained the importance of culturally sensitive CAs to support increased trust and adoption in the context of helping African Americans with chronic diseases and the difficulties of understanding and incorporating cultural aspects into CAs. P10’s focus on creating a multimodal food journal mentioned their participants’ problem of remembering or discovering the voice commands to track their food, and the need to have a better mapping of commands to multiple utterance styles and advanced intent recognition. P13, a project using a self-diagnosis chatbot, reported that their users tended to drop out of the consultation with their chatbot, especially during the early stages, and pointed out the importance of examining and evaluating the mechanisms and approaches that can increase the uptake and utilization of health chatbots. P14 incorporated various strategies into the chatbot to support higher treatment adherence for pediatric patients awaiting a specialist appointment. These included developing a working alliance; having face-to-face communication; using everyday conversational language; and empathic language strategies such as choices, consequences, and nonjudgmental affirmations.

The Evaluation theme encapsulates 3 limitations that authors have encountered during their experiments with CAs: (1) methodological limitations that show challenges in evaluating interactions and performance of CAs; (2) experimental limitations related to the challenges with data collection and analysis and study environment; and (3) lack of guidance in evaluation describing challenges to navigate assessment of CAs without guidance or prior evidence.

By nature, CA interactions are designed to communicate with humans and are dependent on intellectually crafted bidirectional conversations that are hard to generate or replicate. Early efforts toward testing the CAs included scripted conversations to measure the performance of CAs objectively. Yet, scripted conversations fall short in mimicking the actual number of conversations and iterations that may occur in the real world (P1). In addition, a limited number of interactions create a barrier to evaluating the performance effectively. Creating complex scripts or testing with actual patients are some of the solutions proposed (P1).

To provide standard measures to evaluate the performance of CAs, user-CA interactions can be tested in a controlled environment using scripts, such as scenarios, role-playing, or Wizard of Oz testing. However, these scripted or simulated interactions might be limited to providing organically flowing conversations (P1) and accurate assessments due to the simulated nature of system functionalities and user scenarios (P8). Such evaluation yields the results only in a controlled environment, impacting the end user’s judgment (P8). The observations on user-CA interactions may explain a CA’s efficacy to a degree yet not its effectiveness, which can be observed in the real-world environment. Using actual end users with different health and well-being needs in real-world settings and observing them longitudinally may improve the evidence of CA interactions (P1, P11, and P14).

Several projects discussed challenges related to the conversational competence of CAs and the impacts these might have on people’s interactions. Competence here refers to accurately understanding user input and responding appropriately, whether the conversation is an appropriate metaphor for design and how user interactions can be best facilitated, and how CA interactions can be made more accessible and inclusive.

Detecting the topic being discussed can be difficult in several CA scenarios. The unscripted nature of the social talk, often undertaken as part of caregiving interactions, makes it difficult to follow the discussed topics (P3). Even more scripted interactions such as primary care consultations may be nonlinear and fragmented, creating further difficulties in detecting the current topic being discussed (P1). Proposed solutions include advanced topic detection methods (P3) and collaboratively built data sets to support these methods (P1). Using specific phrases to highlight topic shifts during interactions (P1) and personalized systems (P3) may also improve CAs.

Topic detection in CA in health care settings also faces challenges related to specific contexts and users. For example, tracking health care outcomes for different patients and treatments requires an understanding of specific medical terms (P15). Using services such as Amazon Comprehend Medical [63] alongside manual interventions could help expand appropriate vocabulary to improve CA comprehension. Similarly, P12 describes using CAs for people with hearing aids that may require them to use an agreed set of terms or additional supervised training to recognize “audiological intents.”

More generally, there is an ongoing challenge of understanding when someone has finished an utterance while communicating with a CA (endpoint detection). This may require additional research for people with diverse speech patterns such as stammering (P2), to collect the necessary audio data and understand their interactions.

In addition to detecting topics in interaction, CAs in health care interactions face difficulties with ASR. This is also common in CA interactions outside of health care [64]. Audio may be impacted by recording quality ambient noise (eg, other devices such as televisions or other people talking), which could be combated with the use of directional microphones (P1 and P15) [21]. Again, in more nuanced interactions such as using CAs for people with hearing aids, optimal CA responses may require the processing of environmental information such as loudness and signal-to-noise ratio (P12). Inserting such information into the dialog may improve CA performance for this type of interaction.

In addition to audio quality, the language a user produces can also be a limiting factor for CAs. Active error correction by a CA may help in the misunderstanding and nonrecognition of specific terms (P10). For privacy-first interactions, processing of speech data may be performed locally on the device rather than requiring any interaction with servers, though this may reduce the performance of the CA’s speech processing capabilities (P8).

A key challenge in CA design is making the set of possible actions or commands discoverable for users [65]. Understanding and remembering how to interact with CAs in different contexts can create difficulties for people using these systems (P10 and P11). Making CAs open ended and lightweight may allow people to explore systems’ capabilities and adapt them for their purposes (P10). For people with low levels of technology literacy, CA-initiated dialogs can be implemented at consistent intervals to counteract the lack of discoverability of user-initiated features (P11).

CAs rely on turn-taking–based conversational communication and interaction models; however, they may not always be appropriate or require additional scaffolding. General practitioners taking notes using a CA, for example, may need to interrupt interactions with their patients. By contrast, screen-based technologies for the same task can support multimodal, more continuous interactions, and less intrusive data entry (P1). Continuing the multimodal use of information entry for general practitioners may improve or resolve interruptions. Furthermore, monitoring utterances in user-CA interactions may not be as smooth as human-human interactions. Consequently, enabling a CA to detect when someone has finished speaking is critical to more seamless interactions (P2). For diverse speech patterns, such as stammering, we may be able to draw on advice for interacting with various demographics, though crucially, we must understand the nuances of these interactions and design CAs with inclusivity in mind. For scenarios such as filling in forms and surveys with CAs, audio may be an inefficient modality and create design tension with sequential questions and wait times (P17). Optimizing delay usage to limit unnecessary wait times, using shorter phrases, and allowing people to opt-out of audio in multimodal CAs can be used to work against these limitations.

Several projects discuss the need to make CAs accessible to a broad set of demographics or to focus on improving systems to make them inclusive for specific types of people. P14 identifies the need to provide equitable and easy access to CAs that improve health and well-being. They propose allowing access through a web browser and downloadable software, as well as offering technical support and downloadable fact sheets to work with or replace their CA. Off-the-shelf CAs may present difficulties for people in lower-resource areas (eg, limited internet access, financial constraints), which could be improved with systems that use more offline resources (P8), though this may present its own performance challenges that need to be addressed.

Levels of education, experience, and technological literacy can also impact users’ interactions with CAs (P5, P16, and P17). These obstacles may be overcome by providing multiple modalities to compensate for people’s preferences and abilities (P5 and P17) and engaging in participatory design with target demographics (P16). Evaluating such systems may require redeveloping protocols to include people with lower literacy and ensure they are able to understand the questions being posed to them (P17). Specific communities may also require additional thought in designing and evaluating CAs and building on the available sparse research literature (P2 and P9). Working with people who stammer, for example, has seen little research with CA interactions and requires a fundamental understanding of the barriers to successful interactions and how these can be overcome (P2). P9 highlights the need to consider African American communities and how their perceptions towards CA and interactions with them cannot be assumed to be the same as other communities. Consequently, novel methods and evaluation techniques may need to be developed with diverse populations in mind.