Hallucinated Evidence and Real-World Risk—Physicians’ Intent to Act on AI-Generated Abstracts in Emergency Medicine and Trauma Surgery: Cross-Sectional, Vignette-Based Diagnostic Accuracy Study

Introduction

Large language models (LLMs) are neural networks trained with the simple but powerful objective of next-token prediction: given preceding context, they learn a probability distribution over candidate words (or subword tokens) and generate the most plausible continuation [1-3]. Through this autoregressive training on large text corpora—and, in newer systems, paired multimodal data—LLMs can compose, condense, and transform language and, in some settings, interface with allied models for automatic speech recognition, optical character recognition, and even handwriting transcription [4-8]. Beyond fluent text generation, these capabilities enable structured reasoning over documents, retrieval-augmented synthesis, and task automation [9-11].

In health care, early adoption has centered on clinical documentation, patient communication, decision support, and research assistance. Examples include drafting discharge summaries, summarizing extensive chart histories, retrieving and synthesizing evidence, and producing case-specific recommendations. Emergency medicine pilots further explore triage support, imaging protocol suggestions, and real-time protocol reminders [12-19].

Traumatology offers a particularly rich continuum of opportunities for human-in-the-loop LLM support. In the prehospital and emergency department (ED) phases, models could structure triage notes, surface red-flag checklists, and compress prior records for rapid handovers [12,17,20]. During trauma bay assessment, they may provide protocol reminders, role-based task lists, and time-stamped action logs aligned with Advanced Trauma Life Support (ATLS) principles [19,21-24]. For diagnostics, LLMs can suggest appropriate imaging bundles, convert free-text radiology reports into structured injury lists, and automatically detect discrepancies across documentation sources [25-27]. In the operative setting, LLMs might draft preoperative briefs, prepare implant checklists, and generate patient-facing consent documents [28-34]. During inpatient care and rehabilitation, LLMs could assist with daily progress notes, targeted consult requests, discharge instructions, and individualized recovery plans [35-42]. At the systems level, LLMs may help prefill trauma registries, prepare morbidity and mortality reviews, and generate adherence checklists for critical protocols [43-49].

However, the aforementioned benefits are inseparable from risks. LLM outputs can be confidently wrong—“hallucinated”—and may include fabricated facts or citations [50-54]. In clinical contexts, such errors can foster automation bias, degrade diagnostic reasoning, and undermine scientific integrity [52,55,56]. Prior evaluations show that human readers often struggle to distinguish AI-generated prose from human-written prose, blurring lines of acceptable use and intensifying concerns about overtrust [57-60]. Accordingly, any clinical deployment of LLMs must remain strictly human-supervised, be bound to local guidelines and provenance, protect patient data, and be prohibited from autonomous, high-risk decision-making [61-66]. Early clinical integrations should target low-risk, auditable tasks—documentation, summarization, and protocol reminders—where efficiency and usability can be measured without compromising safety.

Against this backdrop of rapidly expanding clinician and patient use of LLMs for medical communication despite well-documented hallucinations and overtrust of AI-generated health advice, a critical unanswered question is not simply whether LLMs can generate persuasive clinical narratives, but whether this persuasiveness can shift clinicians’ diagnostic or management decisions when the underlying content is false [67]. This study was specifically designed to address that gap by evaluating physicians’ willingness to act on LLM-generated misinformation. The primary end point is the proportion of physicians indicating an intent to base clinical decision-making on hallucinated, false evidence embedded in seemingly authoritative text. Secondary end points include physicians’ perceived precision, accuracy, grammatical soundness, and structural quality of the same material. By linking perceived quality to intent to act, our work aims to quantify the diagnostic and decision-making hazards posed by LLM hallucinations and to inform guardrails, training, and deployment policies for safe use in trauma care and beyond.

Methods

Overview

Four research-style abstracts were generated using ChatGPT (GPT-5 Thinking mode; OpenAI) on May 1, 2026, following a standardized, prespecified prompt workflow. Prompts provided the clinical scenario and instructed the model to produce a structured abstract (Background, Methods, Results, and Conclusions; 200-300 words; neutral scientific tone; and no citations). Two abstracts were produced under a true or data-consistent instruction set, restricting content to the provided source information and explicitly prohibiting unsupported additions. Two abstracts were produced under a false or misleading instruction set that required an incorrect yet clinically plausible conclusion and the integration of 2 to 4 predefined false or misleading claims (“hallucinations”), without indicating within the text that deception had been introduced. Given the clinical context, these false claims were selected to represent misinformation with potential for clinically meaningful harm if interpreted as actionable guidance. The complete generation protocol, including the stepwise table, full prompt texts, and condition-specific constraints, is provided in Multimedia Appendix 1.

The table in Multimedia Appendix 2 summarizes, for each abstract, (1) the scenario description, (2) the principal claims, (3) the conclusions presented, and (4) an adjudication of factuality (true vs false) with a brief rationale explaining why the statements are accurate or erroneous. Full, detailed versions of all 4 abstracts are provided in German in Multimedia Appendix 3. A translation of all 4 abstracts is provided in Multimedia Appendix 4.

The 4 abstracts were presented in randomized order within a standardized, paper-based questionnaire. Randomization was computer-generated without replacement to minimize order and carryover effects. The instrument did not disclose the origin (LLM vs human) or the factual status (true vs false) of any abstract.

After reading each abstract, participants completed quality ratings on a 4-point Likert-type scale (1=“poor” and 4=“excellent”) across 4 prespecified domains—precision, factual accuracy, grammatical quality, and structural coherence—and provided an overall quality score. Subsequently, they answered a dichotomous clinical-intent item. To test the hypothesis that participants’ intent to follow a recommendation is chiefly driven by their agreement with the abstract’s conclusion, we elicited—immediately after each abstract—a 2-point agreement rating (1=“disagree” and 2=“agree”) with the concluding recommendation.

We specifically asked clinicians to rate each abstract’s accuracy, precision, grammatical quality, and structural coherence because work in health communication and online health information shows that perceived message accuracy and completeness are core components of credibility, which in turn is a strong predictor of intentions to follow recommended health behaviors [68-70]. At the same time, research on processing fluency demonstrates that texts that are easier to read, that is, grammatically correct, stylistically polished, and coherently structured, are more likely to be judged as true and to elicit greater confidence in subsequent decisions, even when their content is objectively false [71-73]. Finally, clinicians frequently rely on abstracts alone to inform diagnostic and treatment decisions, and variations in abstract reporting quality are known to affect the accuracy of those decisions [74-77]. Together, these literatures suggest that abstracts perceived as more accurate, precise, grammatically polished, and coherent will be seen as more credible and easier to process and are therefore more likely to increase clinicians’ intent to act on the information they contain.

To enhance content validity, participants were explicitly encouraged to propose edits and corrections to the questionnaire items and abstracts. Free-text fields after each abstract and at the end of the survey invited rewording suggestions and the identification of ambiguities, typographical errors, and clinical inaccuracies. All comments were recorded verbatim and reviewed by 2 investigators; minor editorial corrections were implemented for subsequent iterations, while content-altering changes were deferred until completion of the primary analyses.

Participants were physicians working at primary, secondary, and tertiary trauma centers. Specifically, clinicians were recruited from the University Hospital Brandenburg, Charité – Universitätsmedizin Berlin (Campus Benjamin Franklin), the Ernst von Bergmann Clinic, and the Evangelical Hospital Ludwigsfelde in Germany.

Eligible participants were physicians who had completed at least 6 months of training in the ED and were directly involved in acute care, with emphasis on trauma surgery and emergency medicine. Physicians from other specialties (eg, internal medicine) were also eligible if they had participated in the acute management of patients with trauma during their ED training. The full inclusion and exclusion criteria are summarized in Textbox 1.

The primary outcome was the proportion of respondents indicating willingness to follow recommendations in abstracts that contained fabricated (“hallucinated”) claims. Secondary outcomes were domain-specific and overall quality ratings for all abstracts. Per protocol, prespecified attention or comprehension checks were embedded in the questionnaire; responses failing these checks or having substantial missing data were excluded from analysis.

To minimize demand characteristics, participants were masked to the study’s specific hypothesis and to the truth status of the abstracts. The cover explanation stated that the study evaluated the structural and grammatical quality of research abstracts and its association with physicians’ stated willingness to follow the embedded recommendations. Immediately after completing the questionnaire, all participants were debriefed: we disclosed that some abstracts contained intentionally fabricated claims, clarified which recommendations were true vs false, and provided the correct information.

Analyses were performed in SPSS (version 29.0; IBM Corp). Given the preliminary nature of this study and the small sample size, analyses were primarily descriptive and exploratory. The full analysis set included eligible respondents who passed attention checks and completed at least 3 of 4 abstracts. For the binary clinical-intent outcome, we summarized responses using contingency tables and standard diagnostic performance measures, including sensitivity, specificity, positive predictive value, negative predictive value, likelihood ratios, and diagnostic odds ratios, each with 95% CIs. For the 4-point rating outcomes, results are presented as mean (SD) values by group. Exploratory between-group comparisons were performed using 2-sample 1-tailed t tests. Because each physician rated multiple abstracts, these exploratory comparisons did not fully account for within-physician clustering and should therefore be interpreted cautiously as descriptive rather than confirmatory. Missing data were minimal and were handled by complete-case analysis. All reported P values are 2-sided and are presented for exploratory interpretation only.

Because this initial study included only the first 20 physician questionnaires, the present report should be interpreted as a preliminary, exploratory, hypothesis-generating analysis intended to assess instrument feasibility, randomization performance, and the plausibility and direction of observed effects rather than to provide definitive hypothesis-confirming results or broadly generalizable estimates.

Ethical Considerations

This cross-sectional, vignette-based survey enrolled licensed physicians who completed a paper-based questionnaire. Participation was voluntary, limited to adults (aged ≥18 years), and based on written informed consent. Under applicable institutional or national regulations, anonymous survey research involving health care professionals and collecting no directly identifying or patient data does not require formal ethics committee review (78,79). No directly identifying personal data (eg, name, date of birth, or contact details) were collected; responses were recorded anonymously and are reported only in aggregated or anonymized form. To minimize expectancy and demand characteristics, participants were not informed during questionnaire completion of the specific study hypothesis or that some presented abstracts contained deliberate inaccuracies; immediately after participation, all respondents were debriefed, informed of the fabricated elements, and provided with corrected information. No patient data were collected or used. Study data were stored securely with access restricted to the research team and will be deleted or destroyed 5 years after completion of the final analyses and publication. The study was conducted in accordance with the principles of the Declaration of Helsinki.

Results

Approximately 50 physicians were approached for participation. Only 20 (40%) returned completed questionnaires and were included in the final analysis. No returned questionnaires were excluded for failing attention or comprehension checks, having incomplete responses, or failing to meet the prespecified eligibility criteria.

Twenty physicians from several surgical and internal medicine specialties with varying levels of seniority participated in the study. Their specialties, training levels, and clinical and ED experience are summarized in Table 1. Given the small sample, all effect estimates should be interpreted cautiously as preliminary and primarily descriptive.

Each physician rated 4 research abstracts, 2 containing true information and 2 containing false information, resulting in an equal number of ratings for true and false abstracts. Overall, this provided a balanced dataset for comparing responses to correct and incorrect information (Tables 2-5).

For the analysis of diagnostic performance metrics, the 4-point clinical-intent item (“No,” “Probably would not follow,” “Probably would follow,” and “Yes”) was dichotomized a priori into “would not act” (“No” or “Probably would not follow”) and “would act” (“Probably would follow” or “Yes”). Using the true status of the abstract as the reference, physicians’ willingness to clinically act on the information showed greater sensitivity to true abstracts than to false ones.

In other words, physicians tended to act on many abstracts that were true (31/40, 78%), but also on a substantial proportion of those that were false (23/40, 58%), and their decision not to act did not reliably exclude false information.

Perceived accuracy, precision, structure, and grammatical soundness of the abstracts are presented by truth status in Table 4. Ratings for true abstracts were, on average, slightly higher than those for false abstracts for accuracy and precision, while structure and grammar were not significantly different. Formal comparisons did not show statistically significant differences between true and false abstracts on any of the rating dimensions.

Perceived quality ratings displayed a different pattern when abstracts were grouped by physicians’ willingness to act (Table 5). Abstracts that physicians indicated they would act on were consistently rated more favorably for accuracy, precision, and structure than those that they would not act on, whereas ratings of grammatical soundness did not differ significantly.

These findings suggest that physicians’ decisions to act are much more strongly aligned with their subjective quality judgments than with the actual truth status of the abstracts.

CIs were wide for several measures, reflecting limited precision (n=20); results should therefore be interpreted as hypothesis-generating and primarily informative about the direction and plausible magnitude of effects.

Discussion

In this preliminary sample, physicians frequently indicated a willingness to follow recommendations derived from AI-generated medical text even when the underlying content was intentionally false (“hallucinated”). This readiness to act coexisted with consistently favorable ratings of superficial quality—structure, grammar, and perceived precision or accuracy—which suggests that formal features of presentation may unduly influence trust and clinical intent. In other words, persuasive linguistic polish and coherent narrative scaffolding appeared to function as heuristic cues for credibility, potentially overriding the latent uncertainty inherent in LLM outputs. Notably, free-text responses were sparse and provided little insight into respondents’ reasoning, limiting our ability to identify the specific cues (eg, authoritative tone, internal consistency, or familiar phrasing) that triggered endorsement. Taken together, these results are consistent with automation bias dynamics: clinicians may overweight well-formed AI prose and underweight verification, creating a pathway by which hallucinated content could shape decisions. This highlights the need for explicit guardrails (eg, provenance and citation requirements, structured uncertainty displays, and local guideline binding), as well as training that disentangles rhetorical quality from evidentiary quality in AI-assisted clinical communication.

Our findings align with a growing body of work showing that clinicians modify decisions in response to AI guidance—and that the direction of change depends on the accuracy of the advice [80,81]. Prior evaluations of decision support and imaging AI report overall gains when recommendations are correct, but measurable decrements when they are incorrect, consistent with automation bias and anchoring [82]. Recent empirical work illustrates how LLMs can both shape clinicians’ decisions and simultaneously rely on weak or fabricated evidence. In a randomized clinical trial of diagnostic reasoning, Goh et al [83] showed that GPT-4 produced polished differential diagnoses and management rationales that performed at least comparably to those of physicians using standard online resources, underscoring how authoritative such output can appear. A plausible mechanism is processing fluency, whereby text that is easy to read and internally coherent is processed with lower subjective effort, which can be misattributed to truthfulness and credibility (ie, an illusory truth or truth effect) even when objective accuracy is unchanged [84,85]. In a complementary randomized vignette study, Goh et al [86] found that exposure to ChatGPT’s recommendations for patients with chest pain led clinicians to frequently revise their initial triage and treatment decisions, generally improving guideline concordance but clearly demonstrating that chatbot text can sway clinical judgment. At the same time, studies of LLM evidentiary reliability raise serious concerns: in an evaluation of migraine prophylaxis literature, Moskatel and Zhang [87] observed that ChatGPT often offered nuanced-sounding appraisals of drug efficacy while supplying predominantly hallucinated or incorrect references, and Chelli et al [88] likewise reported high hallucination rates and poor reference accuracy when ChatGPT and Bard (now Gemini) were asked to reconstruct systematic review bibliographies. Taken together, these findings suggest that clinicians may be influenced by the rhetorical authority of LLM-generated text even when the underlying evidentiary support is thin or partially fabricated, highlighting the need for careful verification and critical appraisal of chatbot outputs in clinical practice [83,86-88].

Building on these findings, our study adopted a complementary paradigm: we presented clinicians with LLM-style abstracts rather than point-of-care prompts, blinded them both to the provenance of the text (human vs model-generated) and to its veracity, and then quantified the “diagnostic accuracy” of their intent to act by measuring sensitivity to true vs false content. In contrast to prior work by Goh et al [83,86], which examined how ChatGPT’s advice alters concrete diagnostic and management decisions in vignette-based tasks, our design isolates the rhetorical pull of polished language itself, independent of any explicit endorsement of the tool as a decision aid [87,88]. The central result—that coherent, well-written summaries reliably increased clinicians’ willingness to act, even when key claims were false—converges with and helps explain the mixed performance seen in previous LLM decision support experiments: if clinicians are differentially swayed by linguistic fluency rather than evidentiary strength, then LLMs can appear beneficial in some scenarios (where their fluent output happens to align with guidelines) yet propagate error in others (where equally fluent but incorrect content is accepted at face value) [1,89-97].

At the same time, our design addresses gaps left by previous studies. First, we directly link perceived textual quality (precision, grammar, and structure) to intent to act, showing that rhetorical features can function as credibility heuristics independent of content validity [90,93,98]; most prior work has focused on answer correctness rather than the persuasive mechanics of the presentation itself [94-96]. Second, by seeding targeted hallucinations into plausible abstracts, we model a realistic failure mode for LLM-assisted documentation and literature synthesis—contexts in which clinicians routinely encounter long-form prose rather than single-turn suggestions [97,99,100]. Finally, our diagnostic accuracy framing (treating clinician behavior as the index test against the truth of the abstract) provides a transferable metric set—sensitivity, specificity, and balanced accuracy—for comparing interventions aimed at mitigating overtrust (eg, citations by default, uncertainty displays, or guideline binding) [96,101-104]. Taken together, these contributions help reconcile why LLMs can appear “useful” in aggregate yet still produce pockets of harmful influence when they are incorrect—an insight that should inform both deployment guardrails and future trials.

If left unchecked, this pattern could harm clinical practice in several concrete ways. First, persuasive but incorrect AI prose may anchor clinicians to unsafe actions (eg, universally targeting normotension in hemorrhagic trauma or deferring cranial computed tomography [cCT] in anticoagulated head injury), increasing the risk of missed injuries, delayed hemostasis, or preventable deterioration. Second, repeated exposure to fluent but unreliable summaries can erode evidence-based habits, normalizing citation-free assertions and encouraging cognitive off-loading in lieu of verification; over time, this may deskill clinicians in appraisal and differential diagnosis [105,106]. Third, system-level propagation of AI-generated text into notes, order sets, or handovers can amplify errors—a single hallucination can cascade across teams and shifts, inflating unnecessary resource use (imaging, consults, and admissions) or, conversely, delaying indicated interventions [107,108]. Fourth, because LLMs can encode and reproduce training-data biases, overtrust may widen disparities (eg, undertriage of older or marginalized patients if the narrative deemphasizes red flags common in those groups) [109-111]. Finally, reliance on unverified AI recommendations increases medicolegal exposure and undermines a culture of safety: when documentation appears authoritative, deviations from guidelines may go unnoticed until an adverse event occurs, at which point audit trails reveal that decisions were influenced by unverifiable sources [112-114].

The findings should be interpreted in light of the vignette-based design and the outcome assessed. Our primary end point reflects clinicians’ stated intent to rely on or act on information presented in LLM-generated abstracts, which represents a clinically meaningful proximal indicator of downstream impact rather than a direct measure of bedside behavior. Although intent is a recognized predictor of subsequent action, translation into practice may be moderated by contextual factors such as time pressure, team oversight, workflow integration, accountability structures, and opportunities for verification. Accordingly, these results are best viewed as evidence that misleading LLM outputs can plausibly influence decision-relevant judgments under controlled conditions, while underscoring the need for follow-up studies using higher-fidelity approaches (eg, standardized patient encounters, simulation-based assessments, or embedded workflow evaluations) to directly observe behavioral effects and patient-relevant consequences.

This study has several strengths. We used clinically realistic, research-style abstracts in German, randomized their order, and masked participants to both provenance and truth status, allowing us to isolate the persuasive effect of textual features on physicians’ intent to act.

Data quality safeguards (eligibility screening, attention checks, and exclusion rules) further bolstered internal validity, and systematic debriefing upheld ethical transparency. Important limitations temper interpretation. The preliminary sample (n=20) is underpowered for precise estimates and subgroup inferences; the vignette design measures stated intent rather than observed bedside behavior; and sparse free-text responses restricted insight into the specific linguistic cues driving trust. As a pilot study, these findings should be viewed as hypothesis-generating, and replication with larger, more diverse samples is needed to support robust inference and improve generalizability. Generalizability may be limited by the single language and acute care context, by reliance on a single LLM version and prompt specification, and by potential mode effects from paper-based administration. Finally, although SPSS supported our core models, small cluster sizes and software constraints required analytic simplifications for some ordinal and resampling procedures, and unmeasured factors (eg, topic familiarity and local guideline awareness) could still influence responses.

Furthermore, this study has some limitations related to reporting standards. This study applied a diagnostic accuracy framework to physicians’ judgment of LLM-generated abstracts; however, it does not fully adhere to the STARD (Standards for Reporting Diagnostic Accuracy Studies) 2015 reporting guideline. STARD was developed for patient-level clinical diagnostic test accuracy studies involving a defined target condition, patient flow, and real-world testing pathways, which are not present in a vignette-based judgment experiment. Several STARD items (eg, patient recruitment, disease severity distribution, test execution in clinical care, and patient flow diagrams) were therefore not applicable. Nevertheless, we adopted core STARD principles where conceptually appropriate, including an explicit definition of the index test and reference standard, prespecification of accuracy measures, and transparent reporting of sensitivity, specificity, and predictive values. Consequently, the reported diagnostic accuracy metrics should be interpreted as measures of judgment discrimination rather than clinical diagnostic performance. In addition, because we did not perform a formal postsurvey manipulation check to assess whether participants suspected AI authorship or the presence of intentionally false content during completion, we cannot quantify the extent to which such awareness may have influenced judgment criteria and the resulting discrimination metrics. Finally, because exploratory comparisons did not account for repeated ratings within physicians, inferential results should be interpreted cautiously.