Approval was obtained from the institutional review board of the Medical Faculty Ethics Committee of Shanghai Sixth People's Hospital affiliated to Shanghai Jiao Tong University School of Medicine (approval 2024-KY-203), and the study was registered in the Chinese Clinical Trial Registry (ChiCTR2400092800) on November 22, 2024. Informed consent was obtained from all participants. This study was conducted in accordance with the Declaration of Helsinki and relevant ethical guidelines and regulations.

The overall study design and data sources are summarized in Figure 1. This schematic highlights the integration of multicenter retrospective and prospective real-world clinical cases with PubMed case reports, the application of the multiagent framework, and the evaluation of both model performance and human-AI interaction.

We retrospectively screened clinical cases from 2 tertiary care centers: 1228 cases from center A (group A, January 2018-January 2025, tertiary grade A hospital) and 938 cases from center B (group B, May 2018-March 2025, tertiary grade B hospital). All cases were deidentified encounters of patients diagnosed with acute cerebrovascular disease. In addition, 327 stroke case reports were retrieved from PubMed between January 2024 and January 2025 (group C), which predominantly represented diagnostically challenging and clinically complex presentations. For prospective validation, 213 patients were consecutively enrolled at center A between February and May 2025 (group D). Detailed inclusion and exclusion criteria for each group are provided in the Supplementary Material Methods section and Figure S1 in Multimedia Appendix 1.

To ensure patient privacy, all personally identifiable information was removed. Each case was formatted as a single paragraph containing all or a subset of the following elements: patient age and sex, chief complaint, current symptoms, medical history (including illnesses and medications), relevant family history, physical examination findings, laboratory test results, and imaging reports.

To evaluate LLM performance in generating AIS treatment recommendations and assigning stroke subtypes according to the Trial of ORG 10172 in Acute Stroke Treatment (TOAST) classification, each model was tasked with generating a treatment recommendation and the corresponding TOAST classification conclusion. Seven models were tested: Baichuan-M1-14B [18], GPT-OSS-20B [19], Qwen2.5-32B [20], DeepSeek-R1-Distill-Qwen2.5-32B [21], GPT-OSS-120B [19], DeepSeek-R1-671B [21], and GPT-4o (OpenAI; only in group C dataset) [22] (Table S1 in Multimedia Appendix 1). Real-world clinical cases from groups A to D were used for this assessment. Outputs were produced in free-text format without predefined options, reflecting the probabilistic nature of clinical reasoning. To ensure consistent and reproducible evaluation, an automated grader agent was used to quantify accuracy across all LLMs (Supplementary Material Methods section in Multimedia Appendix 1).

All LLMs were evaluated in single-turn interactions using their default parameter configurations, without additional manual tuning. Model inference was performed with the official vLLM framework, providing an optimized environment for efficient large-scale deployment. All models were executed on a cluster of 8 NVIDIA H20-141GB graphics processing units (GPUs) using the official Docker release. For version control, vLLM v0.8.4 was applied to all models except GPT-OSS-20B and GPT-OSS-120B, which were deployed under v0.10.1, thereby ensuring reproducibility and transparency across experiments.

We also evaluated 3 alternative framework compositions: F1 included the Short-CoT agent and the multiple-choice constraint agent but omitted the summary agent; F2 included the guideline-derived long CoT (Long-CoT) agent and the multiple-choice constraint agent but omitted the summary agent; F3 included the summary agent and the Long-CoT agent combined with the multiple-choice constraint agent. This part used a Long-CoT agent to augment the selected LLM decision-making. The Long-CoT involved filtering and restructuring clinical guidelines into a structured decision-making pathway for AIS (refer to the prompt provided in Supplementary Material Methods in Multimedia Appendix 1) [23].

Ground truth for model-level evaluation was defined as the actual treatment decision and corresponding TOAST classification documented in the clinical record, with independent verification for guideline concordance performed by 2 senior stroke clinicians: a neurologist with >10 years of experience and a neurointerventional physician with >15 years of experience. Cases with disagreement or ambiguity regarding guideline concordance were jointly re-evaluated, and eligibility for inclusion was determined by consensus.

The model-level primary outcome was the overall therapeutic and TOAST diagnostic accuracy of LLMs. Accuracy was the case-level proportion correct; TOAST F1 was macroaveraged one-vs-rest F1 across classes, and diagnostic F1 was computed on binary correctness (1/0). Model-level secondary outcomes encompassed safety-related assessments, including instruction adherence, structured harmfulness assessment, and other exploratory safety metrics. For each case, the presence of any omission or hallucination was counted as one event (yes or no), and event rates were computed as events per case (events/total cases) [24,25].

All analyses were performed in Python (version 3.10; Python Software Foundation). Statistical significance was set at a 2-sided P<.05. Binary and continuous variables were summarized descriptively. Accuracy differences between standalone and LLMs augmented by the multiagent framework were tested with the McNemar test, and F₁-score differences were estimated by bootstrap resampling (1000 iterations) with 95% CIs. Paired ordinal outcomes were compared using the Wilcoxon signed-rank test. The ablation study compared the full multiagent framework with predefined variants that removed or substituted key modules (summary agent, guideline-derived Long-CoT vs Short-CoT, and free-form vs constrained multiple-choice outputs) [26]. We fitted physician-case–level binomial-logit generalized linear mixed-effects models (GLMMs), with decision correctness as the binary outcome, and LLM support as the main fixed effect for treatment decision and TOAST classification [27]. Event rates—including omission and hallucination frequencies—and differences between AI-assisted and nonassisted arms were analyzed using chi-square or Fisher exact tests, as appropriate. Outcome metrics and definitions are summarized in Table S2 in Multimedia Appendix 1.

At center A, 1228 cases were screened retrospectively, and 1055 (85.91%) were included in group A. At center B, 938 were screened, and 721 (76.87%) were included in group B. In addition, 213 prospective cases were screened at center A between February and May 2025, of which 161 (75.59%) were included in group D. Of 327 PubMed case reports retrieved, 144 (44.04%) met the inclusion criteria and were included in group C. We finally analyzed 1937 clinical cases (mean age 68.0, SD 13.6 years; n=761, 39.29% women) between January 2018 and May 2025, and 144 PubMed case reports identified from January 2024 to January 2025 (Table 1).

Figure 2A and 2B and Figure S2 in Multimedia Appendix 1 illustrate the simplified system architecture for the standalone LLM and the LLMs augmented by the multiagent framework. Performance varied markedly by model size. Standalone larger-scale LLMs consistently outperformed smaller ones in both treatment recommendation and TOAST classification (Table 2). Additionally, the multiagent framework consistently improved macroaveraged sensitivity and specificity across all evaluated models (Table S3 in Multimedia Appendix 1). For example, across the pooled cohort, GPT-OSS-120B achieved 0.724 accuracy for treatment recommendation, compared with 0.570 for Baichuan-M1-14B and 0.464 for GPT-OSS-20B (all adjusted P<.0001; Table 2). Performance trends were consistent across individual subgroups (Table S4-S7 in Multimedia Appendix 1). Similar gaps were observed for TOAST classification, with DeepSeek-R1 surpassing all smaller models. These results establish model size as a key determinant of baseline performance.

Augmentation with the multiagent framework substantially improved outcomes across all models (Figure 2C-2H and Figure 3A; Figure S3 in Multimedia Appendix 1), with an average improvement of 18.9% compared with standalone LLMs. In group A, Baichuan-M1-14B accuracy increased by 25.8% (F₁-score +15.1%), while DeepSeek-R1-671B improved more modestly (+23.3% accuracy and F₁-score +12.7%). For TOAST classification, GPT-OSS-20B accuracy rose by 39.0% (F₁-score +4.9%), compared with only 2.5% (F₁-score −2.1%) for GPT-OSS-120B. Notably, GPT-OSS-20B exhibited unstable outputs with fluctuating gains. Collectively, these findings show that while model size remains critical for baseline accuracy, the multiagent framework reduces size-related disparities, enabling smaller models to approach the clinical utility of their larger counterparts.

Validation in groups B and C demonstrated consistent performance gains across all groups (Figure 2C-2H and Figure 3A; Tables S4-S7 in Multimedia Appendix 1). The framework-augmented DeepSeek-R1-671B achieved the highest accuracy in both treatment recommendation and TOAST classification. In group C, GPT-4o gained accuracy in treatment recommendation (+14.9%, 0.750 vs 0.653) but showed a decline in TOAST classification (−6.7%, 0.486 vs 0.521), reflecting the predominance of “other causes” and cryptogenic subtypes. The results for alternative framework compositions are shown in Figure 4. Collectively, these findings confirm that the framework provides reliable gains across heterogeneous cohorts.

Generalized linear mixed model analysis in group A confirmed that the multiagent framework was independently associated with higher accuracy in AIS treatment recommendations (odds ratio [OR] 1.72, 95% CI 1.63-1.81; P<.001) and increased selection of appropriate reperfusion strategies (Figure 3B).

In the ablation study (average-only aggregation), treatment recommendation performance improved when using a multiple-choice constraint compared with free-form answer formats (constrained vs free-form: +0.038), and CoT prompting outperformed direct prompting (long vs direct: +0.052; short vs direct: +0.063), suggesting that treatment recommendations benefit from structured answer formats and explicit reasoning scaffolds. Summarization yielded only a modest average gain for treatment recommendation (summary vs no summary: +0.006). In contrast, diagnosis benefited more from summarization (+0.041) and performed better with free-form answer formats (constrained vs free-form: −0.051), motivating task-specific choices of prompting strategy and answer-format control.

Beyond accuracy, we visualized step-by-step reasoning traces for representative success and failure cases to characterize failure modes and deployment-relevant clinical safety risks (Figure 5A and 5B). Because accuracy alone provides an incomplete view of clinical applicability, we further assessed output safety. Compared with the standalone LLM, the multiagent framework achieved higher mean clinical safety scores (4.01, SD 0.35 vs 3.70, SD 0.43) and lower mean hallucination (20.6% vs 33.6%) and omission rates (24.5% vs 38.5%; Figure 5C-5E; Table S8 in Multimedia Appendix 1). Other exploratory safety metrics are shown in Table S9 in Multimedia Appendix 1. These findings demonstrate that structured reasoning may improve reliability and mitigate the risk of unsafe content entering clinical workflows.

Integration of LLM support substantially enhanced physician performance, with the most pronounced gains observed for less-experienced physicians in treatment decisions (0.667 to 0.833 in junior specialists; 0.600 to 0.846 in nonspecialists). On the basis of physician-case decision-level observations (1 observation per physician per case), we fitted a binomial-logit GLMM with decision correctness as the binary outcome, LLM support, physician experience, and specialty as fixed effects, and a random intercept for case and physician. After adjustment, LLM support was associated with higher odds of correct treatment decisions (OR 2.86, 95% CI 2.27‐3.60; P<.0001) and correct TOAST classification (OR 3.63, 95% CI 2.85‐4.64; P<.0001; Figure S4 in Multimedia Appendix 1). The estimated association was strongest among junior and nonspecialist physicians. For treatment decisions, the ORs were 3.96 (95% CI 2.64-5.96) for junior nonspecialists and 3.22 (95% CI 2.13-4.87) for senior nonspecialists; for TOAST classification, the corresponding ORs were 4.66 (95% CI 3.01-7.22) and 4.27 (95% CI 2.74-6.66). Expert specialists showed positive but nonsignificant estimates, consistent with a ceiling effect at higher baseline performance levels (P<.001). Physicians further rated the assistance positively, with a mean score of 3.849 out of 5. Improvements were more modest in senior and specialist physicians (Figure 6).