As this study focuses on the impact of varying types of SC explanations on layperson trust, it is first necessary to understand the purpose and use of explanations in AI. Explanations in theory and practice have long been studied by researchers, resulting in expansive literature on the subject. Here we adopt Lewis’ [21] definition of an explanation as the provision of the causal history of an event. We also draw on Hilton [22] and Miller [23] to argue that explanations are ambiguous in being both a verb and a noun where an explainer explains (verb) something (noun) to generate understanding (verb) in a recipient.

Explanatory theory has typically focused on the function of explanations as a mechanism to transmit information about the explanandum (the event or phenomenon being explained) to further inform a recipient’s knowledge [24]. However, explanations can be used in many ways, including to persuade [25], assign blame [26], or even deceive the recipient [27,28]. Given these purposes, the explainer and the recipient may frequently have differing goals [25].

The goal of the SC explanation is to generate enough trust in the recipient so that they will follow triage instructions to reduce health system burden [8,10]. This is distinct from the goal of the recipient, which is to understand what is causing their symptoms [29].

Explanations may be prevalent in human and social interactions, but they are not ubiquitous. People select when to seek an explanation [26]. Intrinsic provocation to seek an explanation, referred to as explanation seeking curiosity, is strongly predicted by future learning and future utility and moderately predicted by lack of knowledge [30].

The influential information gap theory described by Lowenstein [31] proposes that a delta between an individual’s current knowledge and their desired knowledge cultivates curiosity. It is normatively correct to seek an explanation when an event or phenomenon does not fit one’s mental model [32]. Empirically, information gaps have been shown to provoke explanation seeking in both children [33] and adults [34].

Knowledge and explanation are intimately linked [35]. When presented with information, the recipient must be aware of their lack of knowledge to seek an explanation [31]. An explanation must sufficiently transmit information [36], which is evaluated against prior beliefs and knowledge, particularly in knowledge-rich domains [25], the desired result being an updated mental model in the recipient [26]. The recipient’s perception of an explanation subsequently provides insight into their knowledge of the phenomenon or the event being explained [22,37]. Hence, explanatory quality is critical because “explanations that are good and are satisfying to users enable users to develop a good mental model. In turn, their good mental model will enable them to develop appropriate trust in the AI [18].”

Explanations can take many forms, for example, scientific, causal, teleological, or everyday. A universally accepted taxonomy does not exist [38]. Given this study’s focus on laypeople and prior scholarship suggesting their utility in AI [23,38], our focus will be on everyday explanations. These are a form of explanation, which are commonly observed in social interaction, defined as an answer to a why question [21,23,39].

Lewis [21] asserts that most explanations are typically answers to why questions, for example, “Why did you do that?” or “Why did that happen?” Why questions are implicitly contrastive, with the form of the question being “why that decision rather than something else? [21].” When humans answer a why question, we offer an explanation (P) relative to some other event that did not occur (Q). This is termed a contrastive explanation, where P is referred to as the fact (an event that occurred) and Q is the foil (an event that did not occur) [39].

The factual component of a why question can have many potential foils. Consider the question “Why did you watch Game of Thrones?” Its foils could include “rather than the news?” or “instead of going out?” The foil itself generates context for the explanation to be provided. In human interactions, foils are often not explicitly stated in the why question. Instead, humans infer the foil from the tone and context of the interaction [39]. Importantly, the cause explained is dependent on the questioner’s interest implied through conversation. This further emphasizes the assertion by Miller [23] that explanations are social and conversational [22].

Explaining the contrast can be easier than explaining the fact itself because P does not need to be sufficient for the event to occur, provided it differentiates between the causal difference of P and Q [40]. Contrastive explanations also have the benefit of constraining the information to be provided [39]. The constraining effect of a contrastive explanation is helpful to humans because it reduces the cognitive burden of processing an explanation [26].

Humans rarely provide an exhaustive causal chain as an explanation, preferring instead to select one or two pertinent causes [25,36]. In the example of the television show choice, a layperson would not justify the selection by providing their life story, listing influential childhood events. Instead, the explainer might answer, “It’s more exciting than real life,” and “I was tired,” respectively. These explanations may not entirely explain P; however, they sufficiently and succinctly differentiate between P and Q. For the remainder of this paper, for simplicity, contrastive explanations will be referred to as why-explanations.

Striking an appropriate balance in terms of the complexity of an explanation is difficult. Thagard's [41] theory of explanatory coherence states that people prefer simple, general explanations. This preference has been validated empirically [23]. For example, Read and Marcus-Newhall [42] evaluated this using a scenario about a woman with three symptoms: weight gain, fatigue, and nausea. The study participants received one of the following three explanation types describing what was causing her ill health:

The participants preferred the broad explanation of pregnancy (option 2), preferring simple explanations that had fewer causes and explained more events.

Contemporary laboratory-based studies reiterate human preference for simple explanations over complex ones [13,36,43-45]. However, experiments in natural settings have revealed that complexity increases explanatory satisfaction [28,46] and that complexity preferences are aligned with the complexity of the event itself [47-49].

Frequently, studies in cognitive psychology examine diagnostic explanation through discussion of an alien race’s illnesses to avoid reliance on prior knowledge. This removes a confounder; however, we should be cautious applying the findings to real-life SCs, given the relationship between explanation and knowledge (see Explanation Seeking and Knowledge section). Importantly, most of the experiments required high levels of literacy and comprehension. They were performed on cognitive psychology students or recruits screened for literacy ability. This is in contrast with reality: 1 in 7 of the UK population are functionally illiterate and would struggle to read a medicine label [50], and only half have an undergraduate degree [51].

In short, variance in explanatory preference has been noted between laboratory and natural experimental settings. In addition, experiments have been conducted on study groups with different characteristics to the general population. This reveals a need to validate preferences before generalizing to technologies used by the layperson population.

It should not be assumed that the presentation of explanations in AI systems matches with human explanatory behaviors and needs. Whereas the need for humans to explain themselves is often taken for granted in particular situations, there is continued debate around whether it is necessary for AIs to explain themselves. Turing Award winner Geoffrey Hinton argues that explanations are not necessary because humans cannot explain their own neural processes [52]. A study of medical students supports this because half of them were found to rely on intuitive thinking in diagnostic decision-making [53].

Recognizing that AI systems are frequently used to (help) make impactful decisions, explanations can be said to be required for at least two reasons. First, explanations are necessary for users to adopt AI technologies. When explanations are provided, trust and propensity to rely on systems increases [23,54-57]. If AI systems are not trusted, they are less likely to be adopted by their users, limiting the efficacy of the technology [19,58]. Second, the General Data Protection Regulation, the European Union law for data protection and privacy, places two relevant requirements on AI systems: they (1) must provide “meaningful information about the logic involved in the decision-making process” and (2) should ideally provide “an explanation of the decision [59].”

This results in a focus on explaining the decision-making of AI systems [57]. To do this, the algorithmic processes are examined to generate an explanation (the product), and the contained knowledge is subsequently communicated to the recipient (a social process) [23]. In contrast to human explanatory preferences, current AI explanations predominantly provide answers to how questions, not why questions.

AI models are often treated as black boxes due to their complexity and opacity. The explainable AI field predominantly focuses on increasing the transparency of how models produce their outputs, typically for use by computer programmers and expert users. This essentially answers a how question: “How did you decide that?”

There are a vast number of techniques that are used to explain how an AI model comes to a decision or otherwise produces an output [13]. These functional explanations will be referred to as how-explanations. Two such how-explanations are currently used to explain the outputs of SCs (see Explanations in Symptom Checkers section):

Both these explanatory methods require their human recipient to have domain knowledge to evaluate the explanation. Neither provides an explicitly contrastive explanation or an answer to a why question. Instead, a cognitive burden is placed on a human expert recipient to evaluate and contrast a large number of data points to determine whether they agree with the decision. In the mammogram example, a person not trained in radiology would have insufficient knowledge to understand either explanation, potentially resulting in the recipient perceiving them as poor quality.

When AI systems are deployed to wider society, system builders (such as software engineers and designers) must package the technology into software. For general audiences, the system builders take these how-explanations and translate them into a form comprehensible to a nonexpert [15]. Although this approach seems natural to many computer scientists and lawyers, it is not necessarily the best approach for SCs.

SCs, like many AI systems, are often presented as conversational agents or assistants. Humans are prone to anthropomorphism [62], and virtual assistants have many features that lead users to infer human-like agency in the assistants’ behavior [63]. In human conversation, we prefer social why-explanations (see Everyday Explanations section). If SCs are to act as conversational agents, this suggests that the explanations being offered should adhere to norms of human conversation by creating a shared understanding of the decision made between the system and a human recipient [64]. Given that it is system builders who currently generate explanations for AI systems, the large body of work on explanations rooted in sociological, philosophical, and cognitive theories is underused [38].

The conflict between layperson perceptions of how-explanations and why-explanations is the focus of this study. Given human preferences, the hypothesis is that there will be a preference for why-explanations of SC outputs. As such, a type of why-explanation that has been proposed as more effective and accessible will be included, the counterfactual explanation.

Counterfactual explanations present how the factors considered to reach a decision must change for an alternative decision to be made [26,40,65]. A counterfactual explanation implicitly answers the why question “Why did you decide outcome P rather than outcome Q?” by examining what would happen if the variables V = (v₁, v₂,...) were different [65,66]. Returning to the mammogram example, a user may ask “Why did you diagnose a tumor [rather than no tumor]?” to which the counterfactual explanation may state “If these pixels were not white, I would not have diagnosed a tumor.”

Counterfactual explanations appear naturally in human cognition. They are a pattern that feature prominently in our day-to-day thoughts [67], with the capability to think counterfactually emerging around the age of two [68]. They are also contrastive, aligning with human explanatory preferences (see Everyday Explanations section). Counterfactual explanations are considered an efficient way of communicating causal reasoning [66,69]. They are effective in highlighting model discrimination [70] and, importantly for this work, they offer a type of why-explanation [27,40,66]. Counterfactual explanations do require cognitive effort on the recipient’s part. However, by their nature, they bound the scope of explanation, reducing cognitive burden [69], and are advocated as a more accessible method for nontechnical users [65].

This study aims to investigate layperson trust in SC explanations. Humans have strong preferences for why-explanations, whereas technical methods developed to explain AI systems to date typically give how-explanations. Three explanation types of relevance have been identified for evaluation. To directly address this study’s focus, the current state of explanations in SCs were examined. SCs, like other AI systems, currently tend to provide how-explanations.

Participants were recruited using the web-based platform Prolific. Qualtrics (Qualtrics, LLC) was subsequently used to randomly allocate the participants to a treatment group and gather survey responses. Participants who took less than 210 seconds to complete the experiment were removed as this would imply that they took less than 30 seconds to complete the survey, indicating satisficing. The final data set comprised 750 participants who were paid £1.45 (US $1.96 minimum wage equivalent) for their time.

A number of ethical issues were considered in designing the study, including the following:

To mitigate each of these concerns, nonsymptomatic participants were asked to watch a video of an SC interaction. They were informed that this was not tailored to their own health needs. A General Medical Council–registered primary care physician experienced in SC design advised on the presented materials to ensure medical accuracy, safety, and realism. The participants were limited to individuals residing in the United Kingdom, which offers universal health care, and they were provided with resources to access clinical support if they became concerned. The study received ethical approval (reference: SSHOIIC1A19007) from the University of Oxford.

As exploratory factor analysis generated different components per disease, the participants seemed to have differing conceptualizations of trust. The clean loading of the migraine responses into three components suggests greater nuance in interpretation compared with the less clear factor structure of temporal arteritis. Temporal arteritis did not generate a Depth component, and factor analysis required the removal of a question related to sufficient detail. As indicated previously, temporal arteritis was selected with the expectation that recipients generally knew less about temporal arteritis than about migraine.

Given the link between explanation and understanding [37,85], it follows that a recipient’s perception of explanation quality gives insight into their mental model of the explanandum. The lack of a clear perception of Depth, coupled with the unclear factor loadings, suggests that the participants had less knowledge of temporal arteritis. In other words, there was a larger information gap between participants’ general medical knowledge of temporal arteritis compared with their knowledge of migraine. Where an information gap occurred, an asymmetric knowledge condition was created where the participants had a need for knowledge to be transmitted [63].

Varying the type of explanation provided for migraine resulted in no significant difference in explanatory trust. This is likely attributable to the participants’ knowledge of migraine.

Studies have demonstrated that patients evaluate medical information against their underlying knowledge base [86-88]. It is likely that as the participants watched the symptoms being described in the experiment, they relied upon their existing knowledge and formed a hypothesis of what may be the cause. As migraine symptoms are commonly known, the subsequent diagnosis of migraine aligned with the participants’ knowledge.

An information gap is strongly predictive of an individual’s need to be provided with an explanation [34]. The migraine results are supported by evidence that explanations are only required when there is an information gap between the explanation and existing knowledge [33]. In addition, because migraine is a common disease, it is highly probable; it is known that high priors increase the acceptance of explanations [36]. The lack of effect suggests that for a common, benign, well-known disease, the participants generally did not ask why the SC chose the diagnosis or how it came to the answer. Either the diagnosis made sense, or it did not. Thus, the explanation is not evaluated [14], and there is no impact of the explanation on the users’ trust.

Varying the type of explanation provided for temporal arteritis was reported as marginally significant (P=.09). This claim is made because significant findings when examining the effect of varying the disease (see Impact of Varying Disease section) add further support to the evidence that an effect is being observed. It is likely that the need for an explanation is stronger for temporal arteritis than for migraine. This explanatory need caused the participants to critically evaluate the temporal arteritis explanation. Before exploring these needs, it is instructive to first consider the stimulus that the participants were shown.

The chatbot video ends by informing the user that the disease requires immediate clinical attention and showing the user booking an appointment with a general practitioner. This severity was noted by the participants, 90% (345/383) of whom judged temporal arteritis to be moderately or very severe (Table 2). When presented with a diagnosis of temporal arteritis, most of the participants would not be knowledgeable of its symptoms. A diagnosis of an unknown, severe disease may have surprised or even alarmed the participants, causing an emotional response.

There are two potential factors that stimulated a greater need for explanation in the temporal arteritis group. First, as a rare disease with lesser-known symptoms, temporal arteritis created an information gap that caused a desire for an explanation (see Explanation Seeking and Knowledge section). Second, human emotion is known to affect explanation [37] and can influence how we experience events [89]. Surprise is a known predictor of explanation seeking when evaluating how a stimulus aligns with prior beliefs [34,90]. It is therefore possible that emotions such as surprise or even fear generated by the diagnosis may have provoked a greater need for an explanation. As the measurement of emotion was beyond the scope of this study, this would be a promising direction for future research.

Returning to the main discussion regarding the participants’ need for an explanation, the marginal results suggest that an explanation may have improved trust, but there was no clear effect. The lack of significance was surprising, given the presence of a no explanation experimental treatment. Considering the result in the context of the experimental format illuminates possibilities as to why this should be so.

All experimental treatments, even no explanation, involved viewing a 3-minute chatbot interaction. The conversational nature of the interaction transmitted explanatory information [22]. The participants would have consulted their existing medical knowledge to build a hypothesis of the diagnosis while viewing the video. Finally, the description of the disease itself contained factual information. Therefore, the participants likely viewed all treatments, including no explanation, as a form of explanation. This is despite the no explanation type providing no explicit detail of how or why the SC derived its diagnosis.

The three remaining explanation types provided an answer to a why or how question. When framed by the need to close the temporal arteritis information gap, it seems that these explanations did not fully succeed. It seems that the explanation was not complete enough to be persuasive [45]. Lack of completeness in a medical explanation highlights a serious issue. There are tens of thousands of illnesses and conditions that can affect humans [91]. Of these, a layperson likely knows a handful of common illnesses and those that are the subject of public awareness campaigns. To a layperson, the explanations presented likely did not rule out other diseases that they were aware of. For example, when a layperson evaluated the symptoms presented by the counterfactual explanation, might they have wondered if headache and jaw pain also indicated a stroke? Returning to facts and foils (see Everyday Explanations section), it is logical that individuals can only generate foils of diseases that they are aware of. This is important because currently SCs have no awareness of the specific foil each user generates.

It is possible that the SC explanations are viewed as less effective because of a phenomenon known as causal discounting: “The role of a given cause in producing a given effect is discounted if other plausible causes are present [92].” In this scenario, the cause (temporal arteritis) is discounted in producing the effects (health symptoms) because other diseases could plausibly cause them.

Further support for causal discounting can be found in the text comments left by the participants upon survey completion. Of the 750 participants, 195 (26%) left comments. Of these 195, 59 (30.3%) indicated that they were fearful that a more serious disease may have been missed, equally distributed between the two diseases. This is a common concern, with 55% of patients fearing misdiagnosis [93]. As such, layperson need for reassurance that other serious diseases have been considered and dismissed is an important factor when explaining diagnoses. Although these comments were equally distributed between the two diseases, the need to address causal discounting was greater for the disease that created the larger information gap.

It is possible that the simple explanations provided in this study were not sufficiently complete to close the recipient’s information gap [46] and answer their foil. Although humans may prefer simple explanations [44], they do not blindly prefer them, instead calibrating their preferences according to the context of the explanandum [47]. This study’s findings support those in previous works where explanations that are viewed as incomplete are less satisfying to the recipient [30,48]. In the case of the complex nuances of a diagnosis, this study aligns with the findings that people expect a level of complexity that matches that of the event itself [48]. This is orthogonal to the conversational maxim by Grice [94] that explanations should not suggest causes they think the explainee knows. In an SC scenario, this study suggests that the SC must explain that it has considered diseases that the explainee is aware of.

Having evaluated the effect of varying the explanation type on trust in an explanation for a given disease, this discussion now moves to the second perspective of this study: holding the explanation type constant and varying the disease being explained.

This section builds on the previous finding that the disease being explained is a key factor in determining trust in explanation. As the findings are nuanced, this section sequentially examines the results of each explanation type. Two general themes emerged from this line of enquiry: (1) the uniqueness of symptoms better closes information gaps, and (2) different explanation types prime users to respond according to the explanation’s emphasis. In cases where the emphasis is at odds with the user’s own mental model, this causes cognitive dissonance, highlighting the danger of providing explanations when a user’s explicit foil is not understood.

Before entering this exploration, it should be noted that although the MANOVAs conducted on three of the four explanation types were significant, post hoc t tests were not necessarily so (Table S2 of Multimedia Appendix 2). This suggests that the facets of trust in an explanation combine to create effects. It is also acknowledged that by dividing the respondents into the four explanation type groups, the post hoc tests were underpowered, and larger sample sizes were required for validation.

The most interesting finding when examining results by explanation type are that with regard to input influence, Faith is significantly different between the diseases (P=.053), with the temporal arteritis explanation more trusted than the migraine explanation (Table 5). The explanation given for temporal arteritis states a headache with scalp and jaw pain whereas migraine gives slow onset headache and nausea (Table 1). Returning to the discussion of the simplicity-complexity paradox and causal discounting, the distinctiveness of the combined symptoms given for temporal arteritis likely better closed the participants’ information gap [95]. These unusual symptoms provide a simple explanation that excludes other diseases that the explainee may be aware of, cultivating greater faith [46]. Meanwhile, headache and nausea are symptoms that could be explained by diseases other than migraine, rendering it less trustworthy. As Faith explains the largest amount of variance in the factors of trust, the significance here underscores the marginal result when examining trust by explanation types for temporal arteritis.

Viewing input influence through this lens raises another important point. It is impossible to tell whether the participants perceived the explanation to be a how-explanation or a why-explanation. The aforementioned reasoning indicates that the participants were making a contrastive evaluation (indicating why); yet, we know that an expansion of the input influence explanation would result in a how-explanation (see How-Explanations in AI section).

Social proof is the only explanation format, which is clearly answering how the SC generated a diagnosis. It is known that the provision of an explanation can influence the importance of different features in understanding category membership [96,97]. In addition, laypeople observing AI systems are known to construct an internal mental model of the cognitive process of the software itself [98,99]. Interestingly, question 27—“It’s easy to follow what the system does.”—loaded into Faith, whereas for other explanation types, this question loaded into Comprehension (Table 4). This suggests that the how cue of social proof changed the participants’ perception of the SC’s cognitive mechanism. The participants were implicitly trusting social proof as a clustering technique as opposed to understanding the mechanism itself.

It is surprising that social proof generated a Depth factor, given the nature of this explanation. Return to the conversational nature of the SC causing participants to form a mental model of the disease as they observe questions being answered. By not providing an explanation which contains medical information, the recipient is left to evaluate their mental model against the diagnosis. This raises the question: does providing some form of explanation create more questions than it answers for laypeople?

Comprehension was found to be significant (P=.03), with a small effect size and migraine being better comprehended than temporal arteritis (Table 5). Again, this points to the previous debate around information gaps and existing medical knowledge. The counterfactual explanation for migraine undid nausea, a well-known migraine symptom, matching general knowledge of migraines. For temporal arteritis, suggesting “If your scalp and jaw didn’t hurt, I’d have suggested you had a migraine” did not sufficiently close the information gap for the participants. In the example, the user did have these symptoms. Erasing their existence did not rule out other diseases that the participants were aware of. It is not inconceivable that the participants saw the counterfactual explanation and thought, “But they do have scalp and jaw pain, couldn’t these be caused by another serious condition like a stroke?”

Although counterfactual explanations may be theoretically advocated by social science as being effective in SC-type situations, a good explanation must be “relevant to both the question and to the mental model of the explainee [23,95,100].” The counterfactual explanations provided in this study, although medically valid, did not address the participants’ why questions. This led to a significant impact on comprehension between the diseases; yet, there was no significant impact when comparing explanation types for the same disease. Existing literature calls for the use of counterfactual explanations [38], typically oriented toward expert users. As information gaps are smaller for expert users than for laypeople, the level of information required for transmission may differ. For example, a medical physician would be capable of ruling out serious, common diseases when observing the SC video, whereas the study findings indicate that a layperson may not be able to do so. Again, this stresses the importance of understanding a layperson’s specific foil before generating an explanation.

With regard to the no explanation type, Comprehension was significant (P=.002), with a medium effect size and migraine being comprehended better than temporal arteritis. Notably, the factors generated for no explanation explained approximately 10% more of the variance in data than the factors for other explanation types (Multimedia Appendix 2). The no explanation format did not explain how the SC worked or why it suggested a particular disease. Again, the participants were left to consult their own medical knowledge. These results build on the discussion that providing an explanation may provoke doubt in the recipient by misaligning with their own mental model of both medical knowledge and of how the SC derives results (see Social Proof section).