The study protocol was reviewed and approved by the Ethics Committee of the Cancer Hospital of the Chinese Academy of Medical Sciences (approval 25/331‐5277). The Ethics Committee waived the requirement for informed consent, determining that the retrospective use of deidentified data posed no risk to patients. To ensure confidentiality, all patient identifiers were removed prior to analysis. No images of individual participants are included in this manuscript or its supplementary materials.

Data for this study were obtained from 2 hospitals. Radiology reports were retrieved from the hospitals’ radiological information systems from October 2021 to February 2025, and consecutive cases were collected and comprised 101 liver, 92 lung, and 108 breast cases. Inclusion criteria were as follows: patients (1) aged ≥18 years; (2) who underwent contrast-enhanced liver MRI, chest computed tomography (CT), or breast MRI for disease screening or diagnostic purposes; and (3) who underwent surgical resection with histopathologic confirmation. Exclusion criteria were as follows: (1) lack of relevant preoperative laboratory results and (2) incomplete radiologic findings or impression text. For patients with multiple preoperative examinations, only the most recent scan before surgical resection was selected to ensure the closest temporal proximity to the pathological diagnosis and to maintain the independence of the observations.

We collected patient radiology reports containing both findings and impression sections, along with clinical information such as demographics, chief complaints, and key laboratory test results obtained prior to histopathologic examination. Then, we defined three progressive information input scenarios: (1) basic patient information and imaging findings, (2) scenario A plus the chief complaint or clinical history (only the chief complaint was available for liver and lung cases), and (3) scenario B plus key laboratory test results relevant to each disease. All patient-identifying information was anonymized. We then populated the collected data according to these 3 scenarios and generated separate templates for each disease. To mimic the real-world clinical context, the original text (in Chinese) was kept unchanged during input. Examples of templates for different diseases are provided in Multimedia Appendix 1. All model queries were conducted via the public consumer web interfaces of the evaluated models. These interfaces do not support separate system-level instructions; all role definitions, task descriptions, and formatting constraints were therefore incorporated directly into the user input templates provided in the supplementary materials. All clinical text was manually deidentified prior to input, with patient names, medical record numbers, and institutional identifiers removed. The option to decline data use for model training was selected in the data management settings. No content beyond the deidentified text shown in the appendix templates was entered.

It is important to note that while the LLMs were tested across 3 progressive experimental scenarios to evaluate the incremental value of additional data, the radiologist’s performance remained a fixed historical benchmark and was not retested under the same constrained scenarios.

All radiology reports used as the historical baseline were authored by experienced attending radiologists specializing in abdominal, thoracic, or breast imaging, and detailed demographic and experience characteristics are provided in Multimedia Appendix 2.

All radiology reports and clinical data used in this study were sourced exclusively from the institutional hospital information system with strictly controlled access. None of these records were publicly released or indexed at any time, and therefore, they could not have been included in the pre-training data of any evaluated model.

To minimize bias arising from the models’ context sensitivity and to avoid memory interference, chat sessions were restarted for each model, and responses were collected after each report was typed, yielding a total of 2709 entries.

The LLM-generated reports were exported to Microsoft Excel. A binary semantic matching rubric was used to compare model outputs against the histopathologic reference standard. A case was deemed consistent (score 1) if the core diagnosis matched the reference, regardless of minor stylistic variation; it was deemed inconsistent (score 0) if the diagnosis contradicted the reference or no definitive diagnosis was given. For all models, only the final diagnostic output was evaluated; the internal reasoning traces generated by DeepSeek-R1 were not considered in the scoring process. All cases were independently evaluated by 2 raters: a radiologist (XW, with 23 y of experience) and a medical student (JZ, with 2 y of experience). Disagreements were resolved through consensus with a third evaluator, a radiologist (ZZ, with 21 y of experience). To assess the reliability of the free-text evaluation, interrater agreement between the 2 primary evaluators was measured using Cohen κ coefficient. The κ value and its 95% CI are reported in Table S1 in Multimedia Appendix 3.

For evaluation of the breast cohort, reports that included a specific diagnostic name were classified according to the established biological nature of that entity, irrespective of the Breast Imaging Reporting and Data System (BI-RADS) score. For reports without a specific diagnosis, a strict BI-RADS–based threshold was applied: categories 1 to 3 were defined as benign, and categories 4a to 6 were defined as malignant. For the purpose of determining diagnostic consistency, cases were classified as inconsistent if the original radiology report provided only a risk classification without explicitly stating a specific histologic diagnosis in the impression section. Per the classification rubric described earlier, any case deemed inconsistent was counted as an incorrect diagnosis and was not excluded from the denominator.

We used the 3 generic LLM versions GPT-4o (accessed July 11 to August 9, 2025), Gemini 2.5 Pro (accessed July 17 to August 9, 2025), and DeepSeek-R1 (accessed July 13-31, 2025). All model queries were performed via the public consumer web interfaces of the evaluated models. No specific hyperparameter values were manually set or configured during this process. For reference, the default hyperparameter values reported in the official documentation for each model are listed in Multimedia Appendix 2. The LLMs were tasked with generating radiologic impressions from the input data defined in the various scenarios.

We compared the diagnostic accuracy of LLMs with that of radiologists for “benign-malignant differentiation” and “disease diagnosis.” This evaluation was conducted using two reference standards: (1) for the entire study cohort, histopathologic diagnosis served as the reference standard; and (2) for the breast cohort specifically, the final impressions served as the reference standard to evaluate model performance in BI-RADS classification. We separately recorded the diagnostic accuracy of each model for the benign and malignant subgroups. Furthermore, we adhered to the principle of single-variable analysis to independently assess the effects of model architecture, input scenario, and disease type on diagnostic performance.

Data were analyzed using R software (version 4.4.2; R Foundation for Statistical Computing). To address the imbalanced nature of the cohorts, diagnostic performance for benign-malignant differentiation was evaluated comprehensively using overall accuracy, sensitivity, specificity, positive predictive value, and negative predictive value. Pairwise comparisons of diagnostic accuracy were performed using the McNemar test. To control the cumulative risk of type I errors (false positives) arising from multiple pairwise comparisons, the Holm-Bonferroni correction was applied to all P values derived from the McNemar tests. The correction was applied separately within each diagnostic task for each organ cohort, so that the family-wise error rate was controlled independently for each analysis unit. Statistical significance was defined as an adjusted P<.05.

A total of 301 patients with pathologically confirmed diseases were included in the study (liver: n=101, 33.5%; lung: n=92, 30.6%; and breast: n=108, 35.9%). The flowchart of patient inclusion and exclusion and the diagram of the study design are shown in Figure 1. The mean patient age was 55.1 (SD 12.2) years for the liver cohort, 54.5 (SD 12.3) years for the lung cohort, and 51.1 (SD 11.4) years for the breast cohort. There were 38 (37.6%), 62 (67.4%), and 108 (100.0%) female patients in the liver, lung, and breast cohorts, respectively. Detailed patient demographics and pathologic characteristics are summarized in Table 1.

As shown in Figure 3, the diagnostic performance of the 3 LLMs was compared with that of radiologists within identical input scenarios.

Regarding benign-malignant differentiation, the 3 LLMs consistently achieved their highest accuracy in the liver lesions across all scenarios (range 77.2%‐88.1%; Table 2 and Figure 4A). Regarding disease diagnosis, the lowest accuracy was observed in the breast cohort across all models and scenarios (range 63.9%‐67.6%; Table 3 and Figure 4B).

Diagnostic accuracy and 95% CIs for all cases, stratified by benign and malignant status, are summarized in Table 4. Figures 5 and 6 illustrate model accuracy across these subgroups. Overall, for both benign-malignant differentiation and disease diagnosis, the LLMs demonstrated higher accuracy for malignant lesions (benign-malignant differentiation: range 88.2%‐100%; and disease diagnosis: range 78.9%‐100%) than for benign lesions (benign-malignant differentiation: range 4.2%‐65.6%; disease diagnosis: range 0%‐53.1%). Notably, in the lung cohort diagnosis, GPT-4o achieved 0% accuracy for benign lesions across all 3 scenarios.

Overall, diagnostic accuracy was highest for GPT-4o (range 60.2%‐64.8%), followed by DeepSeek-R1 (range 58.3%‐63.9%) and Gemini 2.5 Pro (range 50%‐55.6%; Table 3, Figure 2C and Figure 3C). A complete matrix of P values for all pairwise comparisons regarding BI-RADS classifications is provided in Table S3 in Multimedia Appendix 3.

While numerous studies have investigated the utility of LLMs for generating the impression section of radiology reports from images or findings [19-22], the diagnostic accuracy of these models across diverse clinical scenarios remains underexplored. This study evaluated the impact of varying clinical input information on the diagnostic accuracy of general-purpose LLMs (DeepSeek-R1, Gemini 2.5 Pro, and GPT-4o) and compared their performance with that of radiologists in real-world diagnostic tasks. While diagnostic accuracy generally improved with the addition of clinical information, this improvement was heterogeneous across models and disease types. In most cases, the addition of clinical history and laboratory results improved the diagnostic accuracy of the model (eg, in the liver cohort, the diagnostic accuracy in scenario C was higher than in scenario A for DeepSeek-R1 [77/101, 76.2%, vs 69/101, 68.3%; unadjusted P=.06, adjusted P>.99], Gemini 2.5 Pro [76/101, 75.2%, vs 69/101, 68.3%; unadjusted P=.10; adjusted P>.99], and GPT-4o [73/101, 72.3%, vs 65/101, 64.4%; unadjusted P=.06; adjusted P>.99]), and this result is consistent with those of Ueda et al [24]. However, we also observed exceptions: In the lung cohort, GPT-4o achieved its highest accuracy in scenario A (68/92, 73.9%). In the breast cohort, the diagnostic accuracy of DeepSeek-R1 peaked in scenario A (73/108, 67.6%) but decreased to 65.7% (71/108) with the addition of laboratory tests; there was no evidence of a difference between the scenarios (adjusted P>.99). This suggests that the addition of laboratory test results has less or even a negative impact on lung and breast diseases in diagnosis. This discrepancy may be because serum tumor markers such as alpha-fetoprotein, carbohydrate antigen 19‐9, and carcinoembryonic antigen play an essential role in the diagnosis of liver diseases [31]. It is also possible that the LLM may misinterpret the addition of extra information as “noise” rather than “signal,” thereby interfering with its diagnostic logic based on image findings and history. However, we emphasize that this is only a descriptive analogy, not a mechanistic claim, and the precise reasons for this performance degradation remain unclear.

Although providing comprehensive clinical information was associated with numerically higher diagnostic accuracy, the improvement was not statistically significant, and overall model performance remained inferior to that of radiologists. Zhang et al [20] reported similar results, noting that even their specialized model struggled to match radiologists’ performance in key dimensions, such as providing a “specific diagnosis” (647/1014, 63.8%). Interestingly, we observed that specific models achieved higher diagnostic accuracy in the breast cohort than radiologists. This discrepancy likely stems from the study’s rigorous evaluation criteria, which classified reports with unspecified case names as BI-RADS categories 0–4b as “inconsistent,” a rule that may have artificially underestimated the radiologists’ performance.

Kim et al [32] reported performance variations across radiologic subspecialties, a finding consistent with our results. Additionally, while the model demonstrated good diagnostic accuracy for common diseases, such as hepatocellular carcinoma and lung adenocarcinoma, a granular analysis of the “other” category revealed poor performance when handling rare and complex lesions. The model exhibited a tendency toward “shortcut learning,” a phenomenon frequently driven by confounding bias [33]. For example, Rueckel et al [34] demonstrated that an AI system designed to detect pneumothorax relied on the presence of thoracic drainage tubes instead of learning features intrinsic to the pathology itself. Similarly, in our study, when hepatic adenoma was misclassified as focal nodular hyperplasia (FNH), the model appeared to treat arterial phase hyperenhancement and persistent enhancement as sufficient criteria for FNH, thereby failing to incorporate a key discriminative feature where the central scar in FNH typically demonstrates delayed-phase hyperintensity. More broadly, the model appeared to rely heavily on combinations of demographic features and coarse imaging patterns, which may bias predictions toward population-level associations rather than individualized assessments. For instance, cases characterized as “young female with persistent enhancement” were frequently predicted as hepatic adenoma, whereas “young male without underlying liver disease but with a central scar” tended to be classified as FNH. A similar pattern was observed in pulmonary diagnoses, where a strong association between “middle-aged female” and lung adenocarcinoma appeared to underpin numerous misclassifications. Although lung adenocarcinoma is indeed more prevalent among women who have never smoked [35], the model’s reliance on statistical regularities may bias predictions toward more common diseases. For example, lung adenocarcinoma and sclerosing pneumocytoma share similar demographic profiles because both are more frequent in middle-aged women. In such cases, dependence on population-level patterns may lead the model to favor the more prevalent diagnosis, resulting in the underrecognition of rarer conditions such as sclerosing pneumocytoma. This limitation is particularly critical in clinical settings where the tolerance for diagnostic error is low [36]. We observed that GPT-4o achieved 0% accuracy for benign lung lesions across all 3 scenarios, indicating that every benign lesion in the lung cohort was classified as malignant. The clinical safety implications of this finding warrant particular attention. In a real-world clinical setting, such systematic overdiagnosis could lead to unnecessary invasive procedures, heightened patient anxiety, and substantial downstream health care costs. This failure mode underscores a broader deployment risk: current general-purpose LLMs may lack the conservative judgment required for low-tolerance clinical tasks, particularly when distinguishing benign from malignant conditions. In addition, automation bias in the model may lead to an imbalanced weighting of multimodal evidence. During diagnostic reasoning, the model may assign disproportionate importance to low-specificity imaging features, such as “possible biliary dilatation,” while failing to adequately integrate critical negative clinical findings and laboratory results. Consistent with our observations, Zack et al [37] reported that aggregate evaluation metrics can obscure underlying biases in individual cases, particularly those relating to demographic attributes.

We also observed performance variations among the models. Results indicated that model performance was consistent across input scenarios for the liver and breast cohorts. Conversely, performance for the lung cohort exhibited greater variability. We hypothesize that this discrepancy stems from differences in imaging modalities, as MRI is a multiparametric and multisequence modality that provides richer information than CT. Zheng et al [21] supported this observation and reported that the odds ratio of MRI versus CT was 3.46, with a 95% CI of 2.56 to 4.67. DeepSeek-R1 demonstrated the highest performance in assessing liver and breast diseases. This finding may be attributable to the use of Chinese-language inputs, given that DeepSeek-R1 was optimized for Chinese text during training, whereas Gemini 2.5 Pro and GPT-4o were not. Prior studies have reported substantial performance variability across languages in general-purpose LLMs. For example, Cozzi et al [38] demonstrated that human-LLM agreement for BI-RADS classification decreased markedly when moving from English to Italian and Dutch, while Meddeb et al [39] showed that translation quality varies significantly across different model-language pairs.

This study has several limitations.

First, a core limitation of this study is the inherent asymmetry in the comparison between LLMs and human radiologists. The experimental design comprised 3 progressively enriched, text-based clinical scenarios specifically constructed for LLM evaluation. In contrast, the historical radiologist impressions used as a comparator were generated in real-world clinical settings, where radiologists likely had access to comprehensive electronic health records at the time of diagnosis, including prior imaging studies, longitudinal clinical documentation, and laboratory data. This imbalance in information availability introduces a significant source of confounding and limits the validity of direct performance comparisons. This study design, while pragmatic for exploring LLM-based diagnostic reasoning, does not support a controlled or fully comparable assessment of radiologist-level performance. Future prospective studies should adopt a symmetrical evaluation framework, in which both LLMs and radiologists are provided with identical inputs under matched conditions. Such designs would enable a more rigorous and unbiased assessment of comparative diagnostic performance. For breast BI-RADS classification, the final radiologist impressions served as the reference standard. This design makes the model versus radiologist comparison structurally circular, as the radiologist defines the correct answer and necessarily achieves 100% accuracy. Model performance for this task should therefore be interpreted as agreement with the radiologist reference standard rather than as a direct comparison of diagnostic capability. Additionally, in the breast cohort, the strict classification rule used to define diagnostic consistency may have artificially lowered radiologist accuracy, thereby lowering the baseline for comparison with the models. As a result, the observed performance gap in this cohort may partly reflect the stringency of the evaluation framework rather than a true difference in diagnostic capability, representing an inherent comparability constraint. This limitation should therefore be considered when interpreting the comparative findings for the breast cohort.

Second, an inherent limitation of this study is that the models evaluated text-based descriptive findings rather than the medical images themselves. Consequently, the diagnostic performance ceiling of the models is entirely dependent on the visual perception and descriptive accuracy of the radiologists who originally authored the reports. If a subtle but diagnostically relevant imaging feature was omitted, understated, or ambiguously phrased in the text, the model necessarily lacked access to that information and could not incorporate it into its diagnostic reasoning. Future studies should develop multimodal model architectures that integrate imaging and text inputs to overcome the inherent constraints of single-modality text-based evaluation.

Third, a limitation concerns the asymmetry in clinical information provided across disease cohorts. In scenario B, breast cases were accompanied by relatively comprehensive clinical histories, including medical, menstrual, and family information. In contrast, liver and lung cases were provided with only the chief complaint. This asymmetry was not due to an inability to retrieve available clinical history, but rather to reflect the absence of equivalently structured data for the liver and lung cohorts in the retrospective data source. This imbalance in contextual information introduces a potential source of confounding and limits the validity of direct cross-organ performance comparisons. While this design partially reflects real-world variability in clinical documentation across organ systems, it nonetheless constitutes a structural inconsistency within the experimental framework. Future prospective studies should adopt standardized clinical information templates across all disease types, ensuring that models are evaluated under matched informational conditions to enable more rigorous and unbiased cross-organ comparisons. Additionally, several aspects of the data input design may have elevated input quality relative to routine clinical practice, thereby limiting external validity. First, in scenario C, we provided only key laboratory markers relevant to each disease type and manually removed the vast majority of laboratory data typically found in real-world electronic health records, effectively and artificially “sanitizing” the dataset. Second, these laboratory values were supplied without accompanying institutional normal reference ranges; because reference intervals vary by assay and equipment, the models may have been forced to rely on generalized reference intervals from pretraining data, potentially introducing uncertainty into the interpretation of laboratory values and contributing to diagnostic error. Furthermore, the exclusion of incomplete or ambiguous radiology reports, for example, 317 cases in the lung cohort, further elevated input quality relative to routine clinical practice. Future studies should use complete, unfiltered laboratory datasets with explicit institutional reference ranges to more accurately assess model robustness under clinically realistic conditions.

Fourth, language represents an unavoidable source of potential bias. As the clinical data were provided in Chinese and different models may exhibit varying degrees of native multilingual optimization, the observed performance differences may be partially driven by differences in linguistic comprehension rather than solely reflecting variations in clinical reasoning ability. Future studies incorporating multilingual inputs, as well as models with different primary language optimization strategies, are warranted to more rigorously disentangle linguistic effects from true diagnostic reasoning performance.

Fifth, the study cohort included only surgically resected cases with histopathological confirmation, which introduces a substantial selection bias. In routine clinical practice, many typical benign lesions (eg, hepatic hemangiomas or breast fibroadenomas) are confidently diagnosed based on imaging alone and therefore do not undergo surgical resection. Consequently, the benign cases in this study are likely enriched for atypical or diagnostically challenging lesions, which may partly explain the relatively low diagnostic accuracy observed for benign diseases. In addition, the statistical power for analyses of benign subgroups was limited, as reflected by wide CIs and unstable estimates. These findings should therefore be interpreted with caution, and the diagnostic performance of the models for benign conditions requires further validation in larger and more representative cohorts.

Sixth, no prospective sample size calculation was performed prior to data collection. According to the STARD (Standards for Reporting of Diagnostic Accuracy Studies) 2015 guidelines (item 14), a priori sample size estimation is recommended to ensure adequate precision and reliability of diagnostic accuracy studies. The absence of such a formal calculation in this study may have affected the precision of the reported performance estimates. Future prospective studies should incorporate predefined sample size calculations to ensure sufficient statistical power and more precise estimation of diagnostic accuracy.

Seventh, it should be noted that the evaluation in this study was based on a strict binary rubric that credited only the definitive final diagnosis. The models were prompted to provide both a final diagnosis and a differential diagnosis; however, the differential diagnosis sections were typically very broad and nondiscriminatory. Crediting a case as correct based solely on the presence of the correct diagnosis anywhere in the output would have artificially inflated the apparent performance. We therefore adopted a conservative approach, while acknowledging that the strict rubric may underestimate the models’ capacity for nuanced clinical reasoning when the correct diagnosis is appropriately prioritized.

Eighth, all clinical text was manually deidentified prior to input, with patient names, medical record numbers, and institutional identifiers removed. However, certain dates embedded in the clinical history, such as the last menstrual period date and other dated clinical events, were not uniformly removed or generalized during the initial data preparation. This represents a limitation in our deidentification protocol and should be acknowledged as a privacy consideration. Future studies using public LLM interfaces should adopt more rigorous deidentification procedures, including the removal or generalization of all date information, to fully comply with best practices for patient data protection.

Finally, the study relied on single-shot prompting, which does not fully capture the probabilistic nature and inherent stochasticity of general-purpose LLMs. As a result, the stability, reproducibility, and intramodel variability of diagnostic outputs could not be systematically evaluated, and the potential variation in reasoning across repeated inference runs remains unquantified. In addition, the prompt templates used in this study were relatively basic and have not undergone extensive optimization. As such, the potential benefits of more advanced prompt engineering strategies were not fully explored in this foundational evaluation. To more rigorously assess diagnostic robustness, future studies should adopt multi-iteration inference protocols. Such designs would enable estimation of output variance, systematic evaluation of hallucination rates, and construction of CIs for model-level performance. Coupling repeated inference with refined prompt optimization strategies will be essential for more comprehensively characterizing the intrinsic reliability of these models in clinical applications.

In conclusion, we evaluated the diagnostic accuracy of DeepSeek-R1, Gemini 2.5 Pro, and GPT-4o in liver, lung, and breast diseases across progressive information input scenarios and compared their accuracy with that of radiologists. A numerical trend toward improvement was observed when systematically increasing the dimensionality of clinical information, but no statistically significant difference was demonstrated. Previous studies have demonstrated the potential of LLMs to automatically extract clinical histories [40,41]. Future clinical AI systems may leverage LLMs to integrate imaging, clinical history, and laboratory data, thereby further improving diagnostic accuracy.