This study was preregistered at Open Science Framework [24]. We programmed the web-based quantitative experimental questionnaire in Qualtrics (Silver Lake) and collected data in April and May 2024. A total of 1828 adult, English-speaking participants were recruited by Dynata [25]. Dynata facilitates data collection for research surveys through its panel of potential web-based participants. We provided Dynata with quotas for age, gender, and race and ethnicity based on the US population composition. Inclusion criteria were at least 18 years of age and residents of the United States. These quotas and inclusion criteria enabled us to receive a diverse range of participant reactions to our experiment.

The study was designed as a 1-way, between-subjects, randomized experiment. Participants assumed the role of a patient advised to undergo an LDCT scan for lung cancer screening. After reading the hypothetical scenario, participants were randomized to receive 1 of 4 results conditions, in which (1) a radiologist alone identified low risk of cancer and recommended a repeat screening CT in 6 months, (2) AI and a radiologist both identified low risk of cancer and recommended a repeat screening CT in 6 months, (3) AI identified medium risk of cancer and recommended immediate additional testing with a nuclear medicine examination (positron emission tomography–computed tomography [PET-CT]) but a radiologist identified low risk and recommended a repeat screening CT in 6 months (ie, radiologist undercalled AI), or (4) AI identified low risk of cancer and recommended a repeat screening CT in 6 months but a radiologist identified medium risk and recommended immediate additional testing (PET-CT; ie, radiologist overcalled AI). We used the randomizer within Qualtrics to randomly assign participants to conditions. Participants read the results both in an electronic health record (EHR) system report and in a transcript of an oral follow-up. Textbox 1 shows the LDCT scan results received in the 4 experimental conditions.

Participants chose their preferred next examination (repeat CT in 6 months or PET-CT now) and rated how much they agreed with the radiologist’s recommendation. They also rated their satisfaction with the radiologist. Participants then answered questions about their attitudes toward medical AI and their MMM preferences. Refer to Multimedia Appendix 1 for the hypothetical scenario and questionnaire measures.

The main dependent variable was a single, 6-point Likert scale measure asking participants how much they agreed or disagreed with the radiologist’s recommendation (1=strongly disagree to 6=strongly agree). We also measured participants’ satisfaction with their radiologist using questions adapted from the Agency for Healthcare Research and Quality’s Consumer Assessment of Healthcare Providers and Systems patient-experience surveys [26]. These included two 10-point scales asking participants how likely they would be to recommend the radiologist to family and friends and how good of a provider they perceived the radiologist.

Covariates included mean composite scores of participants’ attitudes toward medical AI and of participants’ MMM preferences. Attitudes toward medical AI were measured by asking participants to rate their agreement with 7 statements on a 6-point Likert scale (1=strongly disagree to 6=strongly agree). Of these statements, 5 were adapted from previous literature [27,28], and 2 were added by the authors. We reverse-coded statements 2 through 6 before creating the composite so that higher scores represented greater affinity for medical AI. Cronbach α for the 7 measures was 0.62, indicating acceptable internal consistency.

We measured MMM using 2 previously published questions [17,29]. The measures asked, in medical situations, whether participants: (1) tended to lean toward waiting and seeing or taking action (1=I strongly lean toward waiting and seeing to 6=I strongly lean toward taking action) [17], and (2) tended to lean toward doing only what is necessary or everything possible (1=I strongly lean toward doing only what is necessary to 6=I strongly lean toward doing everything possible) [29]. Higher scores on these measures indicated a greater preference for medical maximizing. Cronbach α for the 2 measures was 0.74, indicating good internal consistency. We also categorized participants as minimizers (1-2.5), neutrals (3-4), and maximizers (4.5-6), in line with previous research [29,30], for ease of interpretation.

We conducted ordinary least squares regressions to test whether receiving a recommendation from a radiologist alone, or in agreement or disagreement with AI, affected participants’ agreement with their radiologist’s recommendation. Two-sided P<.05 was considered significant. In exploratory analyses, we also tested how the experimental condition affected participants’ follow-up test choice and participants’ satisfaction with their radiologist.

In sensitivity analyses, we conducted Kruskal-Wallis H tests, Dunn tests, and ordinal logistic regression models to check the robustness of the effects on our main dependent variable, agreement with the radiologist’s recommendation. We also conducted regressions with the inclusion of medical AI attitudes, MMM preferences, and participant demographics as covariates to check the sensitivity of the findings to the inclusion of controls. We ran moderation analyses with participants’ MMM preferences first as a continuous variable and second as a categorical variable.

The study was declared exempt from review by the University of Michigan Health Sciences and Behavioral Sciences institutional review board.

The average time to completion was 8 minutes 32 seconds after excluding 222 speeder participants who completed the questionnaire in 3 minutes or less. A total of 1606 (88%) participants (51% female, mean age 51.49 years) were used in the analyses. Participants in each condition were similar in mean age and sex, indicating successful randomization (Table 1).

In this study, we found that participants penalized radiologists who undercalled AI in favor of less testing but not radiologists who overcalled AI in favor of more testing. Maximizers especially disagreed with radiologists who undercalled AI. However, whereas maximizers agreed with radiologists who overcalled AI, minimizers strongly disagreed. This suggests people’s confidence in the expertise of radiologists who use AI may depend in part on whether their general preferences to maximize or minimize align with the provided recommendations.

Previous research has shown that patients demonstrate algorithm aversion and superior trust in their human clinicians versus AI [8,31-33]. Our research suggests this may not always be the case. When AI and human clinicians are at odds, patients may trust AI over the clinician when AI detects something more than the clinician and recommends more aggressive treatment.

Although radiologists were penalized for undercalling AI, they were not penalized for overcalling AI. This overall attitude toward the radiologist who overcalled AI masked significant variability based on participants’ MMM preferences. Whereas minimizers penalized the radiologist, maximizers rewarded the radiologist. The averaging of these competing attitudes resulted in a level of agreement that did not significantly differ from that in the radiologist-only condition. Thus, radiologists who overcall AI may in fact face significantly lowered trust from patients who have general minimizing preferences.

We did not observe a similar aggregate averaging effect in the radiologist-undercalls-AI condition. This asymmetry may be explained by the perceptions of minimizers. Whereas maximizers demonstrated stable preferences for pursuing more aggressive treatment [30], regardless of whether the radiologist or AI recommended it, minimizers penalized the radiologist who overcalled AI but did not reward the radiologist who undercalled AI.

Minimizers typically prefer not to receive treatment unless considered essential [30,34]. However, previous research has shown minimizers are more responsive to information than maximizers and will change their attitudes in response to evidence [18]. Thus, 1 theory for this asymmetry is that minimizers perceived AI as more accurate than the radiologist and therefore deferred to the AI’s recommendation both when AI recommended more and less testing. This suggests patients, like clinicians, may be prone to automation bias when medical AI is involved [35,36]. Another theory is that the emotional salience of the AI’s cancer results softened minimizers’ attitudes toward more testing. Importantly, patients demonstrated higher agreement in the radiologist-only and radiologist-AI agreement conditions, despite the radiologist recommending less testing just like in the radiologist-undercalls-AI condition. This suggests participants were opposed to the radiologist’s recommendation only when in contrast with the more aggressive AI recommendation. Research suggests the emotional salience of a cancer diagnosis, particularly fear of cancer, influences patients’ decision-making [37-39]. More research is needed to test these theories, including examining patients’ general and context-specific perceptions of the accuracy of AI compared with that of clinicians [31].

Radiologists’ confirmatory “second opinion” of AI did not boost participants’ agreement with them. This finding adds a new dimension to the literature examining patient perceptions of human- versus model-assisted decisions. In 1 study, patients’ trust in AI increased when AI, acting as a second opinion, confirmed the human clinician’s diagnosis, but decreased when the AI was disconfirming [32]. In another study, participants devalued clinicians who consulted a computer-based diagnostic aid for a second opinion, but not clinicians who consulted a human expert [40]. In this study, we found radiologists who agreed with the AI were neither rewarded nor penalized compared with the radiologist alone.

Research shows the human-AI team augments the diagnostic performance of the radiologist [41,42]. However, we show that the human-AI team may not benefit, and could even hurt, the radiologist in terms of patient agreement or satisfaction. This study demonstrated that even small discrepancies (ie, 2 mm) between AI and radiologist interpretations can elicit strong patient reactions. We anticipate larger discrepancies would result in more pronounced negative reactions toward clinicians. As a result, clinicians may feel compelled to overcall or agree with AI to avoid negative patient reactions, as well as potential legal consequences [43,44]. This may lead to incorrect clinician decisions [45], as well as unnecessary testing, biopsies, and procedures. More research is needed to probe clinicians’ attitudes toward the consequences of using AI.

Future research is also needed to better understand patient attitudes toward the human-AI team in various clinical contexts. Research should also interrogate what and how to communicate AI results with patients, especially when the human-AI team disagrees. Offering 2 recommendations to patients may eschew clinicians’ professional agency and burden patients with a choice between following clinician or AI expertise [46]. However, there are ethical considerations to smoothing over discrepancies with a single recommendation that sides with either the clinician or the AI. If both recommendations are medically reasonable, a single clinician recommendation could be seen as too paternalistic [47] and may go against patient preference [48].

This study has several limitations. First, we intentionally recruited a diverse sample of US adults to receive the broadest range of participants; however, this sample is not reflective of the patients who typically would receive LDCT scans. Future research should consider replicating this study with a narrowed sample of participants, particularly smokers, or real patients undergoing cancer screening. For example, we anticipate participants with a higher pretest probability for lung cancer would find a radiologist undercalling AI particularly difficult to accept in this scenario.

Second, we instructed participants that they were eligible to receive an LDCT scan because of “personal risk factors” but did not specify further. Risk factors could range from genetic to environmental to behavioral, and how we develop these risk factors influences our acceptance of test results [49,50]. If participants had different mental models for how they would be eligible for lung cancer screening, these differences could have impacted their responses to the AI and radiologist’s recommendations.

Third, although studies show patients want to know and consent to when AI is used [27,51], we do not yet have a template for how health care systems currently report, will report in the future, or should report AI use to patients. In our study, the EHR results and oral follow-up provided both the AI and radiologist’s interpretations and recommendations. A different presentation of AI’s involvement, such as AI’s interpretation but not a separate recommendation, or simply that AI was consulted but not its interpretation, could elicit different patient perceptions of their radiologist. We also do not provide measures of accuracy for either AI or radiologists. Although radiologists’ accuracy is not typically disclosed to patients who receive imaging, it is possible that patients may inquire about the AI’s accuracy, especially vis-à-vis the radiologists'. How patients understand and interpret measures of accuracy, including more advanced measures, such as sensitivity, specificity, precision, and recall, requires further investigation.

Furthermore, future research is needed to develop validated instruments for measuring attitudes toward medical AI. Although there are validated scales measuring attitudes toward AI in general [52,53], these were insufficient for our experimental context. Thus, the scale we deployed, for which we created a composite score from custom-designed questions and achieved a low Cronbach α, was neither validated nor highly reliable, a limitation of our study. As a result, we may not have adequately controlled for confounding medical AI attitudes, weakening our overall findings.

Finally, the radiologist-patient relationship differs from the classic clinician-patient relationship, as patients might not choose, directly communicate with, or have a preexisting relationship with their radiologist [54,55]. The nature of the radiologist-patient relationship may therefore not generalize to other medical specialties. Despite these limitations, exploring patients’ attitudes toward cancer screening recommendations and the radiologist-patient relationship are understudied yet increasingly important topics in the era of medical AI [56].

As radiologists begin to integrate AI into cancer detection, discrepancies within the human-AI team may influence patients’ reactions toward their radiologist. People may penalize radiologists who undercall AI in lung cancer screening yet may not be more confident in radiologists who overcall or agree with AI. Patients’ MMM preferences moderate this effect and accounting for patients’ MMM preferences may give insight to clinicians into what and how much information to give about AI’s role in their decision-making. Our findings highlight the complexity of the patient-AI-clinician relationship and have implications for clinical practice, communication, and shared decision-making. Future research is needed to determine how radiologists should communicate AI discrepancies to patients in a way that builds trust and maintains their relevance.