We conducted a mixed methods vignette survey to examine how caregivers evaluated AI chatbot responses to pediatric health questions. In this study, a vignette refers to 1 caregiver question presented with 2 alternative chatbot responses in different communication styles, such as brief or formal. Each participant evaluated 3 vignettes (ie, 6 chatbot responses), allowing them to compare responses that varied in tone, length, and information density. For each response, participants rated it on 8 commonly cited evaluation dimensions and their overall satisfaction. We then used a Cumulative Link Mixed Model (CLMM) to examine how ratings on these dimensions were associated with overall satisfaction, while accounting for repeated observations within participants and variation across questions and responses. We also conducted an inductive thematic analysis of participants’ open-ended comments to identify user concerns and potential gaps in existing evaluation frameworks.

This study was reviewed by the University of Southern California Institutional Review Board and was determined to be exempt human subjects research under Study ID UP-25-00074, “Understanding Interactions With Chatbots for Online Medical Information.” All participants were adults who self-reported being the primary caregiver of at least 1 child younger than 18 years. No children were recruited or enrolled as participants. The requirement for informed consent was waived as part of the exempt review determination. Participants were provided with study information before beginning the survey and were informed that participation was voluntary and that they could withdraw from the study at any time. Although survey questions generally required responses to proceed, participants could discontinue participation at any point. To protect participant privacy, survey responses were deidentified before analysis. The study did not collect participant names, contact information, child names, or directly identifiable medical information. Prolific identifiers were used only for participant management, compensation, and quality control purposes, and were converted to study-specific participant IDs before analysis. Access to the data was limited to research team members listed on the approved institutional review board protocol. Results are reported in aggregate, and open-ended responses are presented only as deidentified quotations. Participants were compensated through Prolific. Each participant received at least US $5 for completing the survey, with compensation adjusted when necessary based on actual completion time to meet or exceed Prolific’s minimum rate of US $9 per hour.

We recruited participants across the United States through Prolific, a widely used platform in academic research that allowed us to reach a diverse pool of caregivers with varied ages, educational levels, and professional backgrounds [36,37]. Participants were required to be the primary caregiver of at least 1 child younger than 18 years, have prior experience seeking medical information online, and be a US resident fluent in either English or Spanish. Eligibility was confirmed through an additional screening survey. The survey was first developed in English and then translated into Spanish using GPT-4o. A native Spanish-speaking medical professional at Children’s Hospital Los Angeles reviewed the translation for clarity and medical appropriateness. We also used Gemini 2.0 to back-translate the Spanish version into English as an additional consistency check.

We focused on caregivers seeking advice about common pediatric infectious diseases because conditions such as the flu and ear infections are widespread, making it feasible to recruit participants with relevant firsthand experience. These illnesses also represent a typical chatbot use case: caregivers need quick, reliable guidance to assess symptom severity and decide whether to seek professional care, often under time pressure. This population is therefore both practically accessible and well suited for examining how lay caregivers perceive AI-generated medical guidance in situations that may require timely decisions.

After quality checks and attention screening, our final sample included 191 participants, who together evaluated 573 vignettes and provided 1146 evaluations. The participants represented a diverse demographic, as shown in Table 2.

Figure 2 presents caregivers’ importance ratings across the 8 evaluation dimensions. Since these ratings were collected before participants were exposed to any vignettes with example chatbot responses, they capture respondents’ prior expectations of medical AI.

Accuracy and Credibility were rated as the most important dimensions, with over 80% of respondents assigning them a “High Importance” or “Very High Importance” rating. Usefulness, Thoroughness, Comprehensiveness, and Clarity formed a middle tier, with roughly 50%-70% of caregivers rating them in the top 2 importance levels. Empathy/Warmth received the lowest importance ratings, with nearly half of the respondents rating it as “Low Importance” or “Very Low Importance.” Readability/Style also received mixed feedback, with approximately 25% of respondents rating it in the bottom 2 importance levels.

All dimensions were rated at least “Moderately Important” by a majority of respondents (>50%), indicating that caregivers broadly recognize them as relevant criteria for assessing medical chatbot quality. However, the distribution is far from uniform. Caregivers placed the highest importance on factual qualities such as Accuracy and Credibility, followed by content-oriented qualities such as Usefulness, Clarity, Thoroughness, and Comprehensiveness, and the lowest importance on communicative qualities such as Empathy/Warmth and Readability/Style. As we show in the following sections, however, dimensions that most strongly predict caregivers’ overall satisfaction in practice do not align with this ordering.

Figure 3 shows caregivers’ satisfaction ratings across the 8 dimensions and overall. Ratings clustered toward the positive end of the scale: every dimension received a median of 4 out of 5, and mean scores ranged from 3.77 to 4.26. This concentration at the upper end of the scale suggests that GPT-4o’s responses were generally well received, although it also limits the range of variation available for distinguishing among dimensions.

Although ratings were generally positive, 3 dimensions stood out with more dispersed and more negative scores. Empathy/Warmth was a clear outlier, with the lowest mean score (3.77) and approximately 20% of ratings falling at “Moderately Dissatisfied” or below. Thoroughness (mean 4.01) and Credibility (mean 4.07) also showed wider spread than the remaining dimensions, which clustered between 4.14 and 4.26.

To examine whether these lower and more variable ratings reflected systematic disagreement among caregivers rather than uniform moderate dissatisfaction, we computed Van der Eijk’s agreement score for each dimension. This index quantifies the degree of consensus in a rating distribution on a scale from −1 to +1, with values near 0 indicating disagreement (ie, a uniform spread). We computed the score for each response and then averaged across responses. As shown in Table 3, Empathy/Warmth had the lowest mean agreement (μ=0.378), followed by Thoroughness (μ=0.474), while Accuracy (μ=0.636) and Usefulness (μ=0.548) showed the strongest consensus.

Qualitative comments help explain this pattern. For Empathy/Warmth, participants did not give low ratings because the chatbot failed to produce empathetic responses. Rather, lower ratings mainly reflect participants’ different preferences toward AI communication styles. Some caregivers valued emotional acknowledgment, while others described AI-generated empathy as “artificial,” “unnecessary,” or “patronizing” [P50], and preferred a neutral or clinical tone. As a result, participants who preferred a neutral or professional tone tended to rate empathetic responses poorly.

Tables 4 and 5 report the CLMM results, showing the odds ratio for each evaluation dimension’s association with overall satisfaction, controlling for the other dimensions.

Usefulness was the strongest predictor: each 1-point increase in Usefulness was associated with 2.53 times higher odds of a higher overall satisfaction (95% CI 1.95‐3.27; P<.001). Thoroughness was the second strongest (OR 2.15, 95% CI 1.69‐2.73; P<.001). Accuracy (OR 1.81, 95% CI 1.38‐2.36; P<.001) and Credibility (OR 1.52, 95% CI 1.19‐1.94; P=.002) were also significant predictors but notably weaker than Usefulness and Thoroughness, despite being rated as the 2 most important dimensions in RQ1. In contrast, Readability/Style (OR 1.75, 95% CI 1.43‐2.13; P<.001) and Empathy/Warmth (OR 1.48, 95% CI 1.27‐1.72; P<.001) were both significantly associated with satisfaction despite being rated among the least important dimensions in RQ1. This suggests that presentation-level qualities, that is, how information is worded and formatted, also meaningfully shape caregivers’ evaluations even when caregivers themselves do not see them as priorities.

Comprehensiveness was the only dimension whose effect was not statistically significant (OR 1.05, 95% CI 0.82‐1.33; P=.71). One possible explanation is that its effect is partially absorbed by Thoroughness, which is conceptually adjacent (both concern the amount of information provided, although Thoroughness is about the depth of reasoning while Comprehensiveness is about the breadth and coverage). Alternatively, as caregivers’ open-ended comments suggest (detailed in RQ4), participants may have associated comprehensive responses with excessive length, viewing them as a liability rather than an asset. Participants frequently described preferred responses as “straight to the point” and criticized longer responses as unnecessarily exhaustive, hindering their ability to extract useful information quickly.

Among the random effects, participant-level variance (0.533) was substantial, confirming that caregivers differed meaningfully in their baseline rating tendencies. Question-level variance was smaller (0.074), and response-level variance was near zero (1.66×10⁻¹⁰), suggesting that once the 8 dimension ratings are accounted for, little variation remained across specific response texts.

To check whether results were sensitive to survey language, we fitted the CLMM using only those participants who took the English survey (n=151). The direction and significance of all predictors were identical to those from the full sample, suggesting that the findings were consistent across language groups.

To uncover potential nuances that existing frameworks might have overlooked, we conducted an inductive thematic analysis of 1146 open-ended comments in which participants described what they liked and disliked about each chatbot response and explained their preference. This analysis produced 9 categories of qualities that caregivers desire (Table 6; 443 positive comments) and 9 categories of concerns (Table 7; 415 negative comments). Several categories corresponded to the predefined evaluation dimensions (eg, Credible and Lack of Empathy/Warmth), confirming that the framework captures qualities that caregivers value. However, others surfaced considerations not represented in the existing dimensions, particularly around the type of detail caregivers want, preferences for conversational interaction, and context-dependent reactions to tone and disclaimers. We organize the findings in the subsections that follow around these 4 emergent themes.

Caregivers frequently praised responses for being “straight to the point” (159/443, 35.9%), noting that concise answers helped them identify and “absorb” what mattered more quickly [P127], especially when they were “busy or stressed” [P72] or making decisions under time pressure [P56]. Correspondingly, excessive length was the most frequent complaint (120/415, 28.9%). Many caregivers reported feeling overwhelmed or confused when presented with too much information at once. P28, for example, explained that trying to process a lengthy response while tending a sick child left them “overwhelmed by too much fluff” and thus unable to focus on urgent decisions. P72 put it plainly:

Yet caregivers simultaneously raised concerns about insufficient detail (116/415, 28%). However, closer examination of these comments suggested that “lack of detail” was often used as a catch-all complaint rather than a literal request for longer or more exhaustive answers. When caregivers flagged a response as lacking detail, it was often because a specific type of information felt missing, not that they actually needed deeper or broader explanations. We identified 3 recurring information needs masked behind the complaint of “lack of detail” (Textbox 1):

Some participants pointed toward a resolution: rather than receiving all information at once, they preferred a concise initial answer with the option to go deeper through follow-up questions. P14 explained, “I get my concise answer, and if I felt the need to go deeper, I would proceed as so myself.” Others described frustration at being “thrown into” [P131] answers that tried to “pack everything” [P109] into a single response. P28 made the point most directly: “It’s not like the user only gets one chance to ask a question, they can ask follow-up questions!”

Tone was a prominent theme in open-ended comments. In positive comments, out of 443, 118 (26.6%) praised responses for warmth and a natural, human-like tone. In negative comments, out of 415, 77 (18.6%) criticized responses as too cold, while out of 415, 37 (8.9%) described them as overly empathetic. These opposing complaints reflect the same polarization observed in RQ2, in which Empathy/Warmth had the lowest agreement score among respondents. The qualitative data revealed 3 distinct orientations toward empathetic tone.

Caregivers paid close attention to disclaimers and urgency cues, but their reactions were not uniformly positive or negative. The same language that reassured one caregiver undermined another’s confidence, depending largely on the context in which it appeared.

When responses involved treatment suggestions, dosage guidance, or potentially serious symptoms, caregivers often welcomed clear disclaimers. In these contexts, disclaimers such as “I am not a doctor; please consult a medical professional” were often interpreted as responsible and trust-building. P131 noted that they liked it when a response “suggested this is something that needs to be discussed with a human,” as it prompted them to exercise their own judgment rather than act solely on the chatbot’s advice. For higher-stakes content, a well-placed disclaimer signaled that the chatbot recognized the limits of its role, which participants perceived as “responsible” [P156].

In lower-stakes situations, however, the same language could have the opposite effect. When a response relayed basic facts, routine disclaimers were often seen as redundant, already “assumed” [P37], or even “patronizing” [P149]. Some participants even felt dismissed, as if the chatbot “didn’t have time to take my question seriously” [P61] or had little substantive information to offer: “this small amount of info would infer that...there’s no way to know for sure until you see your healthcare provider” [P37]. P27 similarly described blanket usage of disclaimers as “repetitive,” while P152 noted that they “reduce confidence in the advice” by making every response feel equally uncertain, regardless of actual risk.

Caregivers also wanted responses to communicate urgency more clearly. Several participants felt that the chatbot did not provide enough guidance in distinguishing situations that warranted immediate attention from those appropriate for watchful waiting. P136 noted that a response “should mention when to seek medical help if the issue persists, which could have reassured concerned parents,” and P119 similarly pointed out that caregivers “might be unsure about the severity of certain symptoms” without explicit guidance. At the same time, caregivers were equally wary of responses that leaned toward alarm. They preferred guidance that neither catastrophized, “unlike every other health thing you look up in a Google Search” [P50], nor minimized symptoms but instead helped them calibrate their own response. P28 captured this balance well:

Our findings highlight 3 gaps in how medical chatbots are currently evaluated and designed. First, what caregivers mean by “detail” does not map neatly onto dimensions such as Thoroughness or Comprehensiveness. Existing evaluation frameworks lack a way to capture how efficiently a response delivers what users need in decision-making. Second, existing frameworks evaluate individual responses in isolation but do not assess how well a chatbot supports an ongoing conversation. Third, qualities such as empathetic tone and medical disclaimers are not universally beneficial but depend on both individual preferences and situational context, which current evaluation protocols do not account for.

RQ4 shows that caregivers simultaneously complained about excessive length and insufficient detail, but as our qualitative analysis indicated, these complaints targeted different qualities. When caregivers asked for more “detail,” they were not simply asking for longer, deeper, or more exhaustive explanations. They were asking for responses that better equipped them to make confident decisions. Specifically, they wanted to know what symptoms to watch for, why a recommendation made sense, and where the information came from. Conversely, when they complained about length, they were criticizing the low density of useful information for immediate decision-making.

Existing evaluation dimensions are not designed to capture this distinction. Thoroughness measures the depth of explanation, while Comprehensiveness measures the breadth of topics covered. Neither assesses whether a response delivers the specific information a caregiver needs to take action. The CLMM results support this interpretation: Comprehensiveness had no significant effect on satisfaction (OR 1.05, 95% CI 0.82-1.33), while Usefulness (the dimension closest to actionable guidance in our framework) was the strongest predictor (OR 2.53, 95% CI 1.95-3.27). After accounting for the other dimensions, covering more ground did not, on its own, increase satisfaction; what mattered most was whether responses helped caregivers act confidently. We therefore propose that future evaluation frameworks include such a dimension that captures how efficiently a response delivers the specific information caregivers desire in decision-making, for example, warning signs, concrete next steps, references from authoritative sources, and so on.

A related concern is that some sources caregivers perceived as credible in their comments (eg, WebMD) are not always accurate or the most trustworthy. Yet, current general-purpose chatbots do not systematically vet the sources they draw from, which may include outdated or commercially biased content. The responsibility for source quality cannot rest with the caregiver. Instead, chatbots should actively review and vet sources, directing users to authoritative, peer-reviewed, or government-vetted sources. Emerging platforms such as OpenEvidence [57], which ground clinical responses in peer-reviewed literature, offer a model for adapting such evidence-anchoring strategies for caregiver-facing chatbots.

As reported in RQ4, some caregivers described a natural resolution to the tension between conciseness and detail: start with a short answer and let them ask for more. Rather than passively receiving a dense block of information at once, they preferred to seek exactly what they needed through follow-up questions. This highlights that caregivers do not evaluate a single chatbot response in isolation. A response that delivers all possible information upfront may technically be thorough, but it can frustrate users who want to control the pace and depth of the information they receive.

None of the evaluation frameworks we reviewed assesses this conversational dynamic. Dimensions such as Thoroughness and Comprehensiveness score the content of a single response. They do not measure whether a chatbot helps users navigate a multiturn exchange, for example, by offering to elaborate on severity thresholds or underlying reasoning. As a result, future evaluation protocols should consider how well a chatbot supports multiturn conversations [58,59].

From a design perspective, this suggests that chatbots should treat their initial response as an entry point rather than a comprehensive, stand-alone answer. As multiple caregivers in our sample described, they preferred responses that invited follow-up, for instance, by signaling what additional information was available, such as elaboration on urgency thresholds, reasoning behind a recommendation, or pointers to additional resources. This approach fosters an ongoing exchange where caregivers can efficiently gather what they need without being overwhelmed by information they did not ask for.

RQ3 shows that Empathy/Warmth significantly influenced caregivers’ satisfaction with medical chatbots. At the same time, RQ2 showed that caregivers varied widely in their perceptions of an appropriate tone. The qualitative findings in RQ4 suggest that these differences reflected both personal preferences and situational context, and they challenge a common assumption in chatbot design: that warmer, more human-like communication is universally beneficial [48]. A substantial body of work has focused on building empathetic chatbot personas on the premise that emotional expressiveness builds trust and facilitates human-AI interaction [60-62]. Our data suggest that this holds for some caregivers in some situations but actively backfires for others. Among the 18 caregivers in our sample who rejected empathetic language from AI entirely, warm tone did not just fail to help; it undermined the chatbot’s perceived credibility. These caregivers viewed emotional expressiveness from an AI system as performative or manipulative, suggesting that for a meaningful subset of users, optimizing for warmth across the board risks eroding rather than building trust.

Disclaimers and urgency cues followed a similar context-dependent pattern. Disclaimers were helpful when they signaled the limits of the chatbot’s competence but frustrating when they felt routine or unnecessary, especially in lower-stakes scenarios. Caregivers also valued urgency cues that helped them monitor, triage, and manage symptoms at home but were wary of responses that leaned toward alarm. In both cases, the issue was not the presence of these features but their blanket use across different situational contexts.

The practical implication is the same across all 3 qualities: evaluation frameworks should move beyond treating Empathy/Warmth, disclaimers, and urgency cues as monotonic qualities in which higher is always better. The relevant question is not “How empathetic is this response?” or “Does this response include a disclaimer?” but whether these features are appropriate, given the user and the situation. An empathetic response may be excellent for one caregiver but alienating for another. A disclaimer may build trust for a dosage question and erode it for a question about cold symptoms. Future systems could support this by allowing users to indicate their communication preferences and by adjusting tone and disclaimer use based on the clinical context of the question.

Our findings both validate and extend prior work on the evaluation of medical chatbots. The face validity we observed for established dimensions aligns with the theoretical frameworks proposed by Tam et al [29] and Abbasian et al [32], confirming that these researcher-derived dimensions do resonate with end users. However, our CLMM results show that these dimensions differ substantially in their predictive strength. Usefulness was the strongest predictor of satisfaction (OR 2.53, 95% CI 1.95-3.27), while Comprehensiveness had no significant effect (OR 1.05, 95% CI 0.82-1.33). These findings challenge the implicit assumption in most frameworks that all dimensions contribute comparably to perceived quality and provide empirical evidence on their relative importance. This is an aspect that prior frameworks, largely derived from literature reviews and expert input, have not addressed.

The tension we identified between brevity and detail echoes findings from broader health information-seeking literature [11,63]. Still, our study provides more granular evidence about which specific types of detail caregivers need (actionability, credibility markers, and explanations) versus what they find burdensome (background information and exhaustive coverage). This specificity can guide more targeted response optimization.

Our findings on empathy diverge from a common design assumption in chatbot research: that warmer, more human-like communication is uniformly beneficial. An extensive body of work has focused on building empathetic chatbot personas on the premise that emotional expressiveness builds trust and improves the user experience [49,64,65]. Our data suggest that this assumption does not always hold, as some caregivers reject AI-generated empathy. This aligns with Seitz [66], who found that empathetic expressions in health care chatbots can reduce perceived authenticity, suppressing the trust gains that warmth would otherwise produce. Our qualitative findings converge on a similar point: caregivers who rejected AI empathy tended to perceive it as performative and preferred a professional, information-focused tone instead. The challenge for designers, therefore, is not whether to include empathy but when and how much, and for whom.

The importance of risk-calibrated disclaimers has received limited attention in the medical chatbot literature. Most systems currently apply disclaimers uniformly, likely driven by legal concerns. Our evidence suggests that this practice may be counterproductive, reducing trust in low-risk situations where disclaimers feel patronizing while potentially being insufficient in high-risk situations where more specific warnings are needed. This points toward a need for disclaimer strategies that scale with actual medical risk rather than being applied as legal boilerplate.

Several limitations should be considered when interpreting our findings. First, our study used GPT-4o as the sole source of chatbot responses. While GPT is one of the most widely used AI chatbots, caregivers in practice may encounter AI-generated content through other channels, such as Google search summaries. Evaluations of responses from different models or platforms may yield different patterns. Relatedly, our vignette design presented responses as static text rather than as part of a live conversation. This allowed for controlled comparisons across communication styles but did not capture how caregivers would interact with a chatbot in real time, where follow-up questions and iterative exchanges could shape their experience differently.

Second, our sample, while diverse in age, education, and employment, was predominantly White (132/191, 69.1%) and limited to US-based caregivers. Perceptions of empathy, authority, and an appropriate medical communication style vary across cultural and ethnic groups, and our findings may not generalize to other populations. Similarly, Prolific participants tend to be digitally literate and experienced with online surveys. Caregivers with lower digital health literacy may place greater emphasis on clarity and readability, respond differently to empathetic tone or disclaimers, and be less equipped to evaluate AI-generated content critically. Future work should recruit from clinical settings or community health programs and from more culturally diverse populations to examine how these factors moderate the patterns we observed.

Third, our study focused exclusively on pediatric infectious diseases, which are common, acute conditions. Caregivers managing chronic illnesses or navigating more complex medical decisions may weigh evaluation dimensions differently. We also did not control for contextual factors within the questions themselves. Emergency queries likely elicit different evaluation criteria than routine information requests, yet all questions were analyzed uniformly. Future work should systematically examine how clinical urgency and decision stakes influence the importance of the dimensions.

Fourth, the mixed reactions to empathetic responses raise interpretive questions. We cannot determine from our data alone whether negative reactions to AI empathy stem from perceived insincerity of the specific responses or from a more fundamental resistance to AI expressing emotions at all. A Wizard-of-Oz study, in which participants are unaware whether they are interacting with a human or an AI, would help disentangle these factors.

Fifth, participant eligibility was self-reported through Prolific without independent verification of parental status. However, because pediatric infectious diseases are common conditions familiar to most adults with caregiving experience, this limitation is unlikely to affect our findings meaningfully.

Finally, the pronounced right skew in satisfaction ratings may reflect measurement constraints. Our 5-point scale may have lacked sufficient granularity to capture nuanced differences at the higher end of the distribution. Future studies at larger scales could reveal greater variance in evaluations and enable finer-grained analysis of what distinguishes satisfactory from highly satisfactory responses.

This study provides empirical evidence that established evaluation dimensions differ substantially in their predictive power for caregiver satisfaction with medical AI chatbots. Usefulness was the strongest driver of satisfaction, while Comprehensiveness did not significantly predict satisfaction. Qualitative findings revealed that caregivers want actionable guidance, credible references, and clear reasoning rather than exhaustive detail, and prefer concise initial answers with the option to ask follow-up questions. Our results also challenge the assumption that empathetic tone and medical disclaimers are broadly preferable, as both qualities proved context-dependent, helping in some situations while undermining trust in others. These findings suggest that future evaluation frameworks should empirically weight dimensions rather than treating them as interchangeable, assess how well chatbots support multiturn conversations, and account for individual and situational variation in communication preferences.