This study was conducted and reported in accordance with the Standards for Reporting Diagnostic Accuracy Studies (STARD) guidelines to ensure transparent and complete reporting of diagnostic accuracy.

This study selected representative disease models to evaluate LLM performance across different diagnostic reasoning profiles.

Chronic obstructive pulmonary disease (COPD) was selected to represent common diseases due to its significant public health burden and high incidence. COPD is a highly prevalent internal medicine condition, affecting approximately 380 million people globally and causing over 3 million deaths annually, making it the third leading cause of mortality worldwide. In China, the overall prevalence of COPD among individuals aged 40 and above is 8.6% [30]. Beyond its prevalence, COPD follows well-defined diagnostic criteria, making it a suitable model for assessing LLMs’ ability to recognize high-frequency clinical patterns.

Relapsing polychondritis (RP) was selected to represent rare diseases because of its high disability rate, frequent diagnostic delays, and misdiagnoses in clinical practice, which place a substantial economic burden on both patients and society. RP is an autoimmune disease with an unclear pathological mechanism, affecting multiple organ systems. Consequently, RP was selected as our rare disease case due to these clinical challenges and its significant impact on patients. RP was specifically chosen because its diagnosis requires synthesizing heterogeneous clinical signals across multiple organ systems, making it an appropriate model for testing models’ capacity for multisystem information integration rather than simple pattern matching [31].

The selection of these 2 diseases allowed for a controlled comparison between localized, pattern-consistent conditions and systemic, complex scenarios. Although COPD and RP involve different physiological systems, all cases were developed using a consistent clinical framework, including the chief complaint, history of present illness, physical examination, and key investigations. This approach was intended to keep the diagnostic process comparable across cases by focusing on how clinical information is integrated during reasoning.

The models evaluated in this study are general-purpose LLMs trained on diverse medical corpora rather than specialty-specific datasets. Evaluating their performance across diseases with distinct diagnostic structures, therefore, reflects their baseline reasoning capabilities and reduces the likelihood that results are driven by domain-specific familiarity.

We sourced cases from the China Clinical Case Results Database (CMCR), a national large-scale clinical case results publishing platform funded by the Chinese Association for Science and Technology and constructed by the Journal of the Chinese Medical Association. The database contains peer-reviewed clinical cases of high professional standard (eg, COPD diagnosed per Global Initiative for Chronic Obstructive Lung Disease [GOLD] criteria and RP per McAdam criteria) and provides established reference diagnoses.

Cases of COPD and RP were retrieved between March 31 and April 14, 2025. To ensure suitability for evaluating LLM’s diagnostic performance, cases were selected based on the clarity and clinical consistency of their primary diagnoses. The final diagnosis was required to be either COPD or RP, and cases had to include detailed medical history, physical examination, laboratory tests, and other supporting diagnostic information. Cases with significant comorbidities that could confound diagnostic interpretation were excluded (Figure 1).

To leverage the advantages of LLMs in processing unstructured data, we specifically adapted these clinician-written summaries into a format approximating the authentic language and descriptive style of real-world patients. This involved omitting nonessential sections (eg, titles, treatment discussions, and accompanying tables and videos) and naturalizing the language. Specifically, overly specialized medical expressions were modified to approximate natural patient language, as real-world patients seldom use the professional jargon found in standardized databases (Figure S1 in Multimedia Appendix 1). In addition, this adaptation process also aimed to reduce the likelihood that cases could be directly recognized or matched to memorized training data, thereby mitigating potential bias related to data contamination.

This prespecified and well-defined reference standard reduced potential diagnostic ambiguity during the evaluation phase and enabled consistent comparison of LLM-generated outputs against the ground truth. It also provided a structured basis for applying a rule-based semantic matching approach in the outcome assessment. Furthermore, this standardized case construction ensured that diagnostic reasoning was primarily driven by structured clinical information rather than domain-specific knowledge alone, thereby mitigating potential confounding arising from differences in medical specialties.

As this study was based on retrospective clinical vignettes and did not involve direct patient intervention or participant recruitment, no adverse events occurred during the research process. Baseline demographic characteristics, including patient age and sex, were extracted alongside the clinical information for each case.

We selected 4 Chinese LLMs for evaluation, including 3 general LLMs and one health care-specific LLM. All LLMs are open-source and independently developed in China. The general LLMs selected were Doubao (Doubao 1.5), DeepSeek (DeepSeek-V3), and Kimi (Kimi K1.5), which ranked as the top 3 AI products in terms of Chinese active users in February 2025. The health care–specific model, Leftdoctor GPT (Leftdoctor GPT 3.5), was selected from the “2024 China Healthcare LLMs Top 30” list released on November 5, 2024, by the Chinese Academy of Sciences ('China Internet Week’) and the Center for Informatization Study. All models were accessed through their publicly available interfaces and evaluated under consistent conditions without task-specific adaptation. Models designed for specialized applications, including patient services, medical image analysis, scientific research and innovation, hospital management, and medical record writing, were excluded to ensure comparability across general diagnostic tasks.

To better simulate clinical practice, we provided task prompts to LLMs, requiring them to generate a differential diagnosis based on initial information and then provide a final diagnosis after receiving examination results (Figure 2). As the study was conducted using Chinese LLMs, the original prompts were administered in Chinese (the complete Chinese versions are provided in Multimedia Appendix 1). The English translations of the prompts were:

The evaluation was framed by the deterministic reference diagnoses retrieved from the CMCR database. Since these gold-standard diagnoses were prespecified and verified, the assessment was implemented as a rule-based semantic matching protocol. To maintain consistency, we established predefined synonym boundaries referencing standardized medical vocabularies, such as ICD-10 (International Classification of Diseases, Tenth Revision) or MeSH, in alignment with established medical AI benchmarks. Within this framework, a model-generated diagnosis was recorded as correct if it aligned with the ground truth or its recognized clinical synonyms. The assessment was performed independently by 2 researchers (JYW and XNL) who were blinded to the identity of the LLMs generating the responses, with any discrepancies adjudicated by a third senior evaluator (JY) to reach a final consensus. This structured scoring process yielded a high level of interrater agreement (Cohen kappa=.950).

The diagnostic performance of the LLMs was evaluated using three primary outcome indicators: the rate of final diagnosis within the top 3 DDx list (RTop3D), the rate of final diagnosis as top diagnosis (RTopD), and the rate of final diagnostic accuracy (RFA). RTopD and RTop3D were used to assess the accuracy of the models’ initial diagnostic hypotheses, reflecting their diagnostic breadth and the reliability of the generated candidate list. RFA was defined as the accuracy of the final decision produced after the model integrated supplementary clinical information. By analyzing the transition from initial differential accuracy (RTopD and RTop3D) to RFA, we assessed the models’ clinical reasoning and their capacity to refine diagnostic conclusions as the case progressed. Detailed definitions, assignment standards, and calculation formulas for these metrics are provided in Table S1 in Multimedia Appendix 1.

The mean reciprocal rank (MRR) was used to evaluate the precision of the LLMs in ranking the correct diagnosis. The reciprocal rank for each case was determined by the position of the gold-standard diagnosis among the top 3 differential diagnoses provided by the model. The MRR was calculated using the following formula:

where C corresponds to the number of cases on which the metric is evaluated, and r_i is the rank of the first occurrence of a correct answer in the final list for case i. Specifically, a score of 1, .5, or .33 was assigned if the correct diagnosis appeared as the first, second, or third suggestion, respectively. In any case where the correct diagnosis is ranked beyond the top 3 (r_i>3) or is absent from the list, the contribution to the MRR is set to 0. This metric effectively rewards models that consistently place the gold-standard diagnosis at higher positions within the differential diagnosis list.

To characterize recurrent patterns of diagnostic errors generated by the LLMs, a qualitative analysis was conducted on all cases in which the final diagnosis was incorrect. Following an inductive thematic coding process, 2 independent raters (JYW and XNL) reviewed the reasoning trajectories and iteratively identified five recurrent error domains: (1) failure to recognize specific features, involving omission of critical diagnostic clues or pathognomonic findings explicitly provided in the case description; (2) incorrect attribution, where clinical findings were identified but misinterpreted or assigned to an incorrect etiology; (3) failure in multisystem information integration, characterized by the inability to synthesize manifestations across multiple organ systems into a coherent systemic diagnosis; (4) frequency-based matching bias, referring to the tendency to prioritize high-prevalence diseases despite recognizing disease-specific clues suggestive of a rare condition; and (5) neglect of negative evidence, defined as the failure to incorporate exclusionary findings (eg, negative laboratory results) into differential diagnostic reasoning.

A multilabel coding approach was adopted, allowing a single diagnostic error to be assigned to multiple domains when applicable. Discrepancies between the primary raters were resolved by a third senior evaluator (JY). Given the nonmutually exclusive nature of these error categories, interrater reliability was assessed by decomposing the coding task into 5 independent binary classification problems. The category-specific Cohen kappa coefficients ranged from .824 to .939, with an average value of .898, indicating strong agreement in qualitative categorization.

Several measures were adopted to enhance assessment reliability. Before formal evaluation, the 2 primary raters (JYW and XNL) conducted a calibration session using a representative subset of cases to align interpretations of the scoring criteria and the 5 predefined error domains. All assessments were performed in independent dialogue sessions using previously unused accounts to minimize potential information carryover between cases, and the raters were blinded to the identity of the LLMs generating the outputs, which were anonymized prior to evaluation. Residual disagreements were resolved through blinded adjudication by a third senior evaluator (JY). In addition, screenshots of all model outputs were archived to ensure traceability and allow subsequent verification of the extracted data.

Data were entered and organized in Excel, and statistical analyses were conducted using R software (version 4.6.0; R Foundation for Statistical Computing). Diagnostic accuracy was summarized as counts and percentages. There was no missing data in this study, as all 56 clinical vignettes were complete and all 4 LLMs successfully generated responses for each assigned case. Furthermore, no indeterminate or uninterpretable model outputs were encountered; all LLM responses were determinate and evaluable according to the predefined scoring criteria.

To account for within-case correlation, where each clinical vignette was evaluated by 4 LLMs across 2 diagnostic stages, yielding 448 individual diagnostic responses (the analytic unit), generalized estimating equations (GEE) with a binary logistic link and an exchangeable working correlation structure were fitted using the geepack package in R. An exchangeable structure was chosen because repeated observations within each case lacked a natural temporal order.

The GEE analyses proceeded in 2 stages. First, separate models were constructed for each primary endpoint (RTopD, RTop3D, and RFA). These models included LLM type and disease group (COPD vs RP) as main effects, along with their interaction (LLM × disease), to evaluate performance differences across disease categories. Second, to examine diagnostic refinement, a combined GEE model was fitted by pooling initial and final assessments and adding diagnostic stage as a within-subject factor. This model included the relevant main effects and key interactions (stage × LLM and stage × disease) to quantify changes in diagnostic accuracy. For all GEE models, results are reported as odds ratios (ORs) with 95% CIs, and pairwise comparisons of estimated marginal means were performed using Bonferroni adjustment. To assess the robustness of the primary statistical inferences, sensitivity analyses were conducted by comparing the primary exchangeable working correlation structure with an alternative independent working correlation structure. Consistency of parameter estimates, odds ratios, CIs, and statistical inferences across correlation specifications was examined to evaluate the robustness of the findings to the choice of working correlation structure.

To evaluate diagnostic ranking performance, the MRR was computed for each model as a descriptive summary metric. Given the repeated-measures design, in which the same clinical cases were evaluated across the 4 LLMs, and the nonnormal distribution of the ranking data, statistical comparisons were conducted using the Friedman test based on case-level reciprocal rank scores. When a statistically significant overall difference was detected, post-hoc pairwise comparisons were conducted using the Wilcoxon signed-rank test. To control the family-wise error rate across the 6 possible LLM pairs, the Bonferroni correction was applied (adjusted significance threshold: .008).

A 2-sided P<.05 was considered statistically significant.

This study was conducted in accordance with the protocol approved by the Ethics Committee of Capital Medical University (approval number: 2025SY-166) and adhered to the ethical principles of the Declaration of Helsinki. The research used standardized clinical vignettes derived from a retrospective database for the purpose of benchmarking AI models. In alignment with institutional ethical guidelines for the use of deidentified, retrospective data, all patient information was fully anonymized prior to analysis to ensure that no individuals could be identified. No direct intervention or interaction with human participants occurred during this study. All research procedures were performed strictly following the data privacy and confidentiality standards approved by the institutional review board.

A total of 56 clinical vignettes were evaluated in this study, comprising 28 COPD cases and 28 RP cases. Across all cases, the mean age of the patients was 57.6 (SD 18.7) years, with a sex distribution of 43 males (76.8%) and 13 females (23.2%). In the COPD cohort, the mean age was 69.1 (SD 11.3) years (24 males and 4 females), while in the RP cohort, the mean age was 46.2 (SD 17.8) years (19 males and 9 females).

Across the 3 diagnostic performance metrics, differences in diagnostic accuracy were observed among the LLMs. Compared with Leftdoctor GPT, both DeepSeek and Doubao consistently showed higher diagnostic accuracy across all 3 metrics (RTop3D: OR=4.50, 95% CI 1.71-11.83, P=.002; RTopD: OR=6.16 and OR=7.22, 95% CI 1.62-23.36 and 1.86-28.03, P=.008 and P=.004 for DeepSeek and Doubao, respectively; final diagnosis: OR=6.25 and 9.17, 95% CI 2.55-15.34 and 3.38-24.90 for DeepSeek and Doubao, respectively, all P<.001). In contrast, KIMI did not differ significantly from Leftdoctor GPT for any of the metrics (all P>.05). Overall, DeepSeek and Doubao demonstrated better diagnostic performance, whereas the performance of KIMI was comparable to that of Leftdoctor GPT (Table 1).

Significant differences in diagnostic accuracy were observed between the 2 diseases across all 3 metrics. Compared with RP, diagnostic accuracy was substantially higher for COPD for RTop3D (OR=50; 95% CI 10.11 -247.23; P<.001), RTopD (OR=108.33; 95% CI 16.67 -703.98; P<.001), and RFA (OR=11.50; 95% CI 3.24 -40.86; P<.001). These findings indicate that the LLMs achieved markedly better diagnostic performance for COPD cases than for RP cases (Table 1).

The incorporation of additional clinical information led to an overall upward trend in diagnostic accuracy, although the difference between the top-1 differential diagnosis and final assessment stages was of borderline statistical significance (P=.05) (Table 1). Furthermore, no significant interaction was observed between diagnostic stage and LLM type (all P>.05), indicating that the models responded similarly to the supplementary data.

Despite the lack of a strictly significant interaction between diagnostic stage and disease type (P=.07), descriptive results revealed distinct performance trajectories between the 2 disease categories (Table 2). For COPD, all LLMs already achieved high diagnostic accuracy early in the top-1 differential diagnosis stage (82%‐93%), with minimal further improvement after additional clinical information was supplied. In contrast, diagnostic accuracy for RP was substantially lower at the top-1 differential diagnosis stage (7%‐36%) but improved markedly by the final assessment stage (25%‐79%). These findings suggest that the overall improvement across diagnostic stages was primarily driven by marked gains in RP accuracy, whereas diagnostic performance for COPD remained consistently high throughout the diagnostic process.

A significant interaction between disease type and LLM was observed for all diagnostic metrics, including RTop3D (Wald χ²₃=9.42; P=.02), RTopD (Wald χ²₃=8.52; P=.04), and RFA (Wald χ²₃=18.62; P<.001), indicating that differences in LLM performance varied across disease.

Parameter estimates from the GEE model indicated that this interaction was mainly associated with Doubao. Specifically, the interaction between Doubao and COPD was statistically significant across all three diagnostic metrics: the RTop3D metric (OR=.12, 95% CI .03-.56; P=.006) and the RTopD metric (OR=.08, 95% CI .01-.46; P=.005), as well as for the RFA metric (OR=.20, 95% CI .06-.71; P=.01). In contrast, the interaction terms for DeepSeek and KIMI were not statistically significant (all P>.05).

These findings suggest that the variation in diagnostic performance across diseases was mainly attributable to disease-specific performance differences in the Doubao model.

Ranking performance, which accounts for both the correctness and rank position of differential diagnoses, was evaluated using MRR.

For COPD diagnosis, DeepSeek, KIMI, and Leftdoctor GPT achieved the highest MRR scores (.89 (SD .32)), followed by Doubao (mean .82, SD .39). Given the repeated-measures design of the evaluation across the same clinical cases, the Friedman test was used. The analysis indicated no statistically significant difference in diagnostic ranking performance among the 4 LLMs (χ²₃=1.385; P=.71; Table S2 in Multimedia Appendix 1).

For RP diagnosis, diagnostic ranking performance differed notably across the LLMs. Doubao achieved the highest MRR score (.39, SD .48), followed by DeepSeek (.37, SD .46), KIMI (.14, SD .32), and Leftdoctor GPT (.10, SD .28). Given the repeated-measures design, a Friedman test was conducted based on case-level performance scores, revealing a statistically significant overall difference among the models (χ²₃=21.933; P<.001). Post-hoc pairwise comparisons using the Wilcoxon signed-rank test, also performed at the case level, with Bonferroni correction (adjusted significance threshold of α=.0083 for 6 comparisons), indicated that both Doubao and DeepSeek significantly outperformed Leftdoctor GPT (both unadjusted P=.002). Furthermore, Doubao significantly outperformed KIMI (unadjusted P=.008). Although DeepSeek scored higher than KIMI, this difference did not reach the adjusted significance threshold after Bonferroni correction (unadjusted P=.01). Differences between all other LLM pairs were not statistically significant (Figure 3, Table S3 in Multimedia Appendix 1).

Distinct error patterns were observed between COPD and RP cases (Table 3). Among the 12 incorrect COPD diagnoses, errors were limited to failure to recognize specific features (10, 83.3%) and incorrect attribution (2, 16.7%), with no concurrent error types identified.

Among the 55 incorrect RP diagnoses, a total of 61 error events were identified, with 6 cases (10.9%) involving multiple error types. The most common errors were neglect of negative evidence (21, 34.4%) and failure to recognize specific features (14, 23%), followed by incorrect attribution (9, 14.8%), frequency-based matching bias (9, 14.8%), and failure in multisystem information integration (8, 13.1%).

Model-specific differences were also observed. Leftdoctor GPT showed the highest number of errors (21), most commonly neglect of negative evidence (10). Kimi demonstrated the broadest distribution of error categories (21), whereas DeepSeek errors were mainly related to neglect of negative evidence and failure to recognize specific features (5 each). Doubao showed the fewest errors (6), without a dominant error category.

Sensitivity analyses were conducted using alternative independent working correlation structures. The resulting parameter estimates, ORs, CIs, and statistical inferences were materially consistent with those obtained under the primary exchangeable structure, indicating that the study findings were robust to the choice of working correlation structure (Multimedia Appendix 2).

This study evaluated the diagnostic performance of 4 Chinese LLMs across common and rare diseases using a stepwise, hypothetico-deductive framework. Three main findings emerged. First, all models demonstrated substantially higher diagnostic accuracy for the common disease (COPD) compared with the rare disease (RP). Second, incremental clinical information improved diagnostic accuracy primarily in rare disease scenarios, with considerable variation across models. Third, differences between models were minimal under relatively straightforward diagnostic conditions but became more pronounced in diagnostically challenging cases.

A prominent finding of this study is the consistent performance gap between common and rare diseases across all evaluated models. Similar patterns have been reported in recent studies, where LLM performance declines in complex or rare disease scenarios [32]. While this phenomenon is often attributed to differences in data availability [33,34], our results suggest that it may also relate to the distinct clinical structures of these conditions and the influence of learned associations on model outputs.

For common diseases such as COPD, clinical presentations tend to align with frequently encountered and well-represented patterns in training data. Qualitative observations show that errors in the COPD group were largely confined to the failure to identify specific features, while instances of incorrect attribution were infrequent across multiple evaluations. This suggests that when key diagnostic information is correctly recognized, models can generally map it to the intended diagnosis. In these situations, LLMs can rely on strong probabilistic associations to produce accurate and stable diagnostic suggestions [35].

The disparity between COPD and RP extends to differences in organ system involvement and diagnostic breadth. While COPD is primarily localized to the respiratory system, RP is a systemic condition characterized by multiorgan involvement and a complex symptom profile [25]. This complexity requires the synthesis of heterogeneous signals, a task where LLMs showed limitations. Our analysis identified a higher diversity of error types in RP cases, including the neglect of negative evidence and failures in multisystem integration. The presence of frequency-based matching bias, where models prioritized high-prevalence conditions despite identifying RP-specific cues, indicates that a reliance on learned associations may affect the rigorous refinement required for complex, multisystemic conditions [36].

Consequently, their performance remains less stable in low-prevalence or diagnostically complex scenarios, which has important implications for their safe clinical application [37,38].

This study also provides insight into how LLMs respond to sequentially provided clinical information. While additional data had a limited impact on diagnostic accuracy for COPD, it significantly improved performance in RP cases for some models. This asymmetry suggests that the use of incremental information depends on the level of initial diagnostic certainty [39]. In the COPD group, the high initial diagnostic accuracy across models left limited room for further improvement through incremental data. Although infrequent diagnostic errors occurred in the final assessment stage, such as the omission of specific features in either the history or laboratory results, the strong initial signals associated with common conditions allowed LLMs to maintain stable performance, largely unaffected by the addition of new information.

In contrast, when initial diagnostic signals are weak or ambiguous, as in rare diseases, the introduction of new information can meaningfully shift the probability distribution and improve diagnostic accuracy [40]. However, the qualitative observations indicate that while incremental information facilitates performance gains, it does not fully resolve the reasoning challenges inherent in complex diagnoses. Even with access to comprehensive laboratory and imaging results, LLMs still exhibit systematic flaws in the final reasoning stage, most notably the neglect of negative evidence [36]. This suggests that diagnostic failures in rare disease scenarios are linked not only to information scarcity but also to difficulties in the logical processing of exclusionary findings. Therefore, simply increasing the volume of information input may be insufficient to overcome the underlying reasoning patterns that limit LLM performance in complex diagnostic tasks.

Another important observation is that differences between models were relatively small in COPD but became more pronounced in RP. This suggests that model performance may converge under conditions where diagnostic patterns are clear and well-represented, but diverge when cases require handling uncertainty or integrating less typical information [41].

In relatively straightforward scenarios, most models are able to generate similar high-probability outputs, resulting in comparable performance. However, in more complex or ambiguous cases, models may differ in how they prioritize competing diagnostic possibilities or respond to incomplete information. These differences may reflect variations in training data composition, model architecture, or alignment strategies [42].

Qualitative observations provide preliminary insights into these model-specific patterns. In RP diagnosis, Leftdoctor GPT exhibited a higher frequency of errors, particularly in the neglect of negative evidence, which may indicate challenges in integrating exclusionary information. Kimi showed a broader distribution across error categories, whereas DeepSeek’s errors were more concentrated in feature identification and the neglect of negative evidence. In contrast, Doubao achieved the highest diagnostic accuracy in the RP group with the fewest total errors and no dominant error type, suggesting a relatively balanced performance across different reasoning domains. It is important to note that, given the limited number of error events in this qualitative analysis, these observations should be regarded as preliminary rather than definitive conclusions about model capabilities. Further validation in larger-scale studies is required to confirm these behavioral patterns.

The findings of this study suggest that the clinical use of LLMs may be context-dependent. While these models provide relatively reliable diagnostic support for common diseases with typical presentations, their outputs in rare or atypical cases require careful interpretation due to lower stability and higher intermodel variability.

From a clinical perspective, the identified error patterns offer practical guidance for diagnostic assistance. Clinicians should specifically verify whether AI-generated suggestions have appropriately integrated all exclusionary findings to ensure that sufficient diagnostic breadth is maintained for complex presentations. From a research perspective, these results underscore the importance of evaluation frameworks that reflect real-world, stepwise clinical reasoning. Such approaches may provide more realistic estimates of model performance compared to single-step evaluations based on complete information. Additionally, the preliminary error profiles identified in this study can guide targeted improvements in diagnostic capabilities. Future work could explore specialized prompting strategies or fine-tuning methods to help models weigh negative evidence and integrate multisystem clinical information more effectively. Beyond diagnostic reasoning, advancing Chinese LLMs will require expanding their applications across diverse medical specialties and complex decision-making scenarios. Integrating multimodal data such as medical imaging, pathology, and genetic information may help address the instability and information dependence observed in rare disease reasoning [43]. Specialty-specific optimization and cross-disease validation are critical for enhancing generalizability and reliability, while assessment frameworks that incorporate error patterns can provide a more comprehensive measure of clinical reasoning than accuracy metrics alone. Because demographic information such as age and sex was preserved in the clinical vignettes, the observed diagnostic performance may partially reflect how LLMs use demographic cues during diagnostic reasoning. Future studies should further investigate the extent to which demographic characteristics influence diagnostic outputs and whether demographic-related biases exist across different models.

This study has several limitations. First, the sample size was relatively small, leading to sparse-data instability in some statistical estimates, as reflected by large ORs with wide CIs. Although the sample size is consistent with prior exploratory studies of LLM diagnostic performance, larger multicenter studies are warranted to validate these findings across more diverse clinical scenarios [32]. Second, all models were tested using a standardized single prompt, which ensured fair comparisons but may not reflect their performance under alternative prompting strategies. In addition, each prompt was evaluated in a single run, which does not allow assessment of stochastic variability in model outputs; multiple runs could have better quantified or mitigated random fluctuations inherent to LLM-based systems. Third, our clinical vignettes were curated to exclude major comorbidities and included only confirmed cases with complete information. While we naturalized the language to reflect real patient narratives rather than textbook-style descriptions, this design may yield performance estimates that are higher than those observed in unselected real-world clinical practice. Fourth, while the vignettes were paraphrased, the use of public datasets for benchmarking involves an inherent, unquantified risk of semantic contamination. Since language models operate on semantic embeddings rather than relying solely on exact string matches, text modification may not entirely preclude the recognition of cases encountered during training. Finally, our model selection represents a snapshot based on benchmark rankings during the study period and does not encompass all representative Chinese LLMs. Given the rapid evolution of AI technology, performance rankings may shift as new models are released, and future research should incorporate a broader, more up-to-date range of models to provide a more comprehensive evaluation of their evolving capabilities.

This study evaluated the diagnostic capabilities of 4 Chinese LLMs for common (COPD) and rare (RP) diseases using a step-by-step, clinical vignette-based approach. LLMs demonstrated strong diagnostic performance for common diseases but substantially lower and more variable performance for rare diseases. This performance disparity reflects the interplay between training data prevalence and the inherent challenges of synthesizing multisystemic clinical information. Although stepwise clinical information improved accuracy in rare disease cases, it did not eliminate the underlying reasoning limitations, particularly in integrating information and weighing evidence, which remained even when complete data were available. These findings suggest that current LLMs may perform well in pattern-consistent conditions but remain limited in handling diagnostic uncertainty and complexity. Careful evaluation and appropriate clinical oversight are therefore essential for their safe application in practice.