This study was approved by the institutional review board of The First Affiliated Hospital of Wenzhou Medical University (approval number KY2025-R084). The requirement for written informed consent was waived because of the retrospective design. All patient identifiers were removed before the reports were provided to the LLMs and human reviewers.

A total of 4480 Chinese radiology reports with modification records were obtained from the radiology system of our hospital between January 2023 and June 2024. Reports needing image re-evaluation to verify errors, diagnoses revised due to misdiagnosis, and controversial reports were excluded, resulting in a final corpus of 1363 reports containing 1551 errors (X-ray: 27, computed tomography [CT]: 1250, and magnetic resonance imaging [MRI]: 86). The dataset covers various anatomical parts of the body from different imaging modalities. The errors in the reports were all generated by radiologists during routine clinical work. The dataset was randomly divided into a test set (n=1263) and an internal validation set (n=100). Additionally, 100 error-free reports were added to the internal validation set. For further validation of the generalizability of our approach, 200 English reports from the Medical Information Mart for Intensive Care (MIMIC-III) dataset were used, including 100 reports with 148 inserted errors and 100 error-free reports (see Note S1 in Multimedia Appendix 1).

Errors in the reports were initially annotated by a junior radiologist and checked by basic LLMs (eg, DeepSeek-R1) to find potential errors (details are provided in Note S2 in Multimedia Appendix 1). The basic LLMs participated only as static annotation auxiliary tools, and their model parameters remained frozen throughout the entire process without any form of training or fine-tuning. All interactive data generated during the annotation phase were strictly limited to the inference level and did not affect model parameter updates through gradient feedback. LLMs served only as supplementary screening aids to flag potential candidate errors and did not determine the final reference standard. All annotations were subsequently reviewed by a senior radiologist who retained the authority to reject the suggestions of LLMs, thereby ensuring comprehensive error detection while minimizing the introduction of model bias. The report reviewed by the senior radiologist was the final annotated data. Errors were categorized into 5 types based on the characteristics of the Chinese language and previous studies [11]: (1) omission, (2) addition, (3) semantic error, (4) location discrepancy, and (5) others. The detailed definition of each type of error is provided in Table S1 of Multimedia Appendix 1. The severity of the errors was categorized as either “clinically significant” or “not clinically significant” in accordance with recent studies [12]. Clinically significant errors were considered to be of such magnitude that they could alter the meaning of the report, thereby risking misinterpretation by the clinician.

To enhance the error detection ability of LLMs, we randomly selected 100 reports from the test set for multiple iterations of model optimization. First, the LLMs were assigned the role of a professional radiologist and informed that their core task was to detect errors in radiology reports. The detection scope was then restricted using the 5 error types and enhanced by integrating chain-of-thought reasoning [17] along with a few-shot examples. In addition to thinking step by step, the model was further instructed to “split reports for sentence-by-sentence review” and to verify anatomical sites and medical terminology based on different categories. A true positive was defined as when the LLMs flagged a text segment that corresponded to errors in the reference standard (details are provided in Note S3 of Multimedia Appendix 1). To mitigate the “hallucinations” [8] phenomenon caused by LLMs’ automatic correction of diagnostic content, a clear restriction—“prohibiting modification of the original report contents”—was implemented. By using these instructions and restrictions, the accuracy of the model’s error recognition at the word level was effectively improved. The prompt templates were continuously adjusted and improved during the experimental exploration process until the model no longer showed significant improvement. The temperature was set to 0.3 based on a subset, ensuring stability and high accuracy. Our prompt template and parameters are provided in Note S4 of Multimedia Appendix 1. All of the LLMs were run once per report, with calling processes using a “single report, single conversation, no context accumulation” approach. This design ensured that the knowledge representation ability of LLMs remained unchanged, fundamentally avoiding the potential impact of data leakage on subsequent test set evaluation results.

To validate the application of LLMs on real clinical data, 2 experiments were conducted (Figure 1). In part 1, 8 widely used LLMs were evaluated for error detection on the test set. These included international mainstream models such as GPT-4, GPT-4o (OpenAI), Gemini-1.5-Pro (Google), Claude-3.5-Sonnet (Anthropic), and models primarily trained on Chinese corpora such as DeepSeek-V3, DeepSeek-R1 (DeepSeek), Qwen-Plus (Alibaba Cloud), and GLM-4 (Zhiyuan AI). The model’s output included the analysis, error fragment, and revision. The error detection rate was used to evaluate the performance of the models, and the time consumed was recorded from prompt submission to final response. To facilitate quality management of error detection, the LLMs’ ability to classify error types was also evaluated. A retrieval-augmented generation [18] and in-context learning [19] framework was applied, retrieving the top 3 most relevant examples per test sample to perform error-type classification based on predefined categories. This task serves as an indirect evaluation of the overall improvement in the model’s semantic understanding and reasoning abilities resulting from the proposed optimization.

In part 2, the top 2 LLMs were further analyzed on an internal validation set and an external validation set. Eight human readers from various backgrounds—including 2 senior and junior radiologists, 2 nonradiologists, and 2 nonphysicians—were compared with the LLMs in terms of error detection performance and time consumption on the internal validation set. The analysis and correction results provided by the LLMs were evaluated by an experienced radiologist. Detailed information about the human readers is provided in Note S5 of Multimedia Appendix 1.

Statistical analyses were performed using Python (version 3.8.19) and the pandas library (version 2.0.3, NumFOCUS). A paired test was used to assess differences between the LLMs and human readers. P values <.05 were considered statistically significant. Additionally, 95% CIs were calculated using the Wilson method. The model’s error detection performance was evaluated by detection rate, while its classification performance was assessed through precision, recall, and F₁-score. All statistical analyses were performed at the error level, with each error treated as an independent observation rather than at the report level.

The study flowchart is shown in Figure 2. The test set comprised 1263 reports containing 1431 errors, including 552 omission errors, 183 addition errors, 339 semantic errors, 202 location discrepancy errors, and 155 categorized as others. Clinically significant errors outnumbered not clinically significant errors (808 vs 623).

On the internal validation set (Table 2), DeepSeek-R1 achieved a detection rate of 83% (100/120, 95% CI 76%-89%), and Claude-3.5-Sonnet achieved a detection rate of 80% (96/120, 95% CI 72%-86%). Except for semantic errors, where DeepSeek-R1 had significantly higher detection rates (25/27, 93% vs 20/27, 74%; P=.01), no statistically significant differences in detection rates were observed between DeepSeek-R1 and Claude-3.5-Sonnet in the overall and subgroup analyses (P=.27-.99; Tables 3-5). DeepSeek-R1 had a comparable FPR with Claude-3.5-Sonnet (3% vs 4%; P>.99). On the external validation set, DeepSeek-R1 achieved a detection rate of 94% (139/148, 95% CI 89%-97%), and Claude-3.5-Sonnet achieved a detection rate of 93% (137/148, 95% CI 88%-97%). DeepSeek-R1 had a lower FPR compared to Claude-3.5-Sonnet (5% vs 4%; P>.99).

In the overall analysis, there were no statistically significant differences in the average performance in detection rate between DeepSeek-R1 and radiologists (DeepSeek-R1, junior radiologists, and senior radiologists: 83%, 95% CI 76%-89% vs 80%, 95% CI 72%-86% and 78%, 95% CI 70%-85%; P=.39 and P=.19, respectively). DeepSeek-R1 had a significantly higher average detection rate than nonradiologists and nonphysicians (83%, 95% CI 76%-89% vs 66%, 95% CI 57%-74% and 38%, 95% CI 30%-47%; P<.001, respectively; Table 3).

In the subgroup analysis of different imaging modalities, there were no statistically significant differences in the detection rates of X-ray or MRI and CT reports between DeepSeek-R1 and radiologists (P=.07-.99). The detection rate of DeepSeek-R1 was significantly higher than the average detection rate of nonradiologists average in CT reports (83%, 95% CI 75%‐89% vs 67%, 95% CI 57%‐75%; P=.002) and nonphysicians average in CT and X-ray or MRI reports (83%, 95% CI 75%‐89% vs 22%, 95% CI 9%‐45% and 41%, 95% CI 32%‐51%; P<.001 and P<.001, respectively; Table 3).

The performance of DeepSeek-R1 in detecting addition errors was significantly worse than that of the best-performing radiologist (53%, 95% CI 32%-73% vs 89%, 95% CI 69%-97%; P=.03; Table 4). The performance of DeepSeek-R1 was significantly higher than that of nonradiologists in detecting semantic errors (93%, 95% CI 77%-98% vs 59%, 95% CI 41%-75%; P<.001) and significantly higher than that of nonphysicians in detecting omission, semantic, and location discrepancy errors (78%, 95% CI 61%-89% vs 20%, 95% CI 9%-35%; P=.01; 93%, 95% CI 77%-98% vs 49%, 95% CI 28%-63%; P<.001; 97%, 95% CI 83%-99% vs 48%, 95% CI 31%-66%; P<.001; Table 4). There were no statistically significant differences in detection rates in the analysis of error severity between DeepSeek-R1 and radiologists (P=.11-.99). The performance of DeepSeek-R1 was significantly higher than that of nonradiologists and nonphysicians in detecting not clinically significant errors (91%, 95% CI 78%-97% vs 57%, 95% CI 41%-72% and 53%, 95% CI 36%-67%; P<.001 and P<.001, respectively), and significantly higher than nonphysicians in detecting clinically significant errors (80%, 95% CI 70%-87% vs 32%, 95% CI 23%-42%; P<.001; Table 5).

The FPR did not show statistically significant differences between DeepSeek-R1 and radiologists (DeepSeek-R1 vs senior radiologists and junior radiologists: 3% vs 0% and 1%; P=.25 and P=.61, respectively). DeepSeek-R1 demonstrated a significantly lower FPR than nonradiologists and nonphysicians (3% vs 13% and 17%; P=.02 and P=.002, respectively; Figure 4A).

The total reading time of DeepSeek-R1 was 4.62 hours, significantly longer than that of the slowest nonphysician (4.62 h vs 3.6 h; P=.01). The total reading time of Claude-3.5-Sonnet was 0.44 hours, significantly shorter than that of the fastest nonradiologists (0.44 h vs 1.56 h; P<.001; Figure 4B).

The correction accuracy of DeepSeek-R1 and Claude-3.5-Sonnet was 95% and 91%, respectively (Figure 5). The correction accuracy of DeepSeek-R1 in location discrepancy and other errors, as well as the correction accuracy of Claude-3.5-Sonnet in omission and addition errors, were all 100%.

This study systematically evaluated the ability of mainstream LLMs to detect and correct errors in Chinese radiology reports using a large-scale, real clinical scenario dataset. Enhanced by domain-specific prompts, chain-of-thought reasoning, and strict output constraints, DeepSeek-R1 achieved detection rates of 89% (95% CI 87%-90%), 83% (95% CI 76%-89%), and 94% (95% CI 89%-97%) across test, internal, and external validation sets, respectively, matching radiologists and outperforming nonradiologists and nonphysicians. Moreover, DeepSeek-R1 and Claude-3.5-Sonnet also corrected 95% and 91% of flagged errors, respectively. These findings offer new perspectives on the localization and application of LLMs for automated quality control in digital health care settings.

Our study expanded the application of LLMs to non-English environments by using the largest error-labeled Chinese radiology report dataset (1363 reports, 1551 real errors). Our prompt engineering strategy, integrating chain-of-thought reasoning and strict output constraints, was crucial in mitigating hallucinations and focusing models on detection, thereby enhancing their reliability for clinical integration [9,17]. We found that DeepSeek-R1 achieved an 89% (95% CI 87%-90%) detection rate on the test set and 83% (95% CI 76%-89%) on the internal validation, translating to a 25% to 30% absolute gain over the 52% zero-shot ceiling reported by Yan et al [20] for Claude-3.5-Sonnet in Chinese ultrasound reports. Moreover, DeepSeek-R1 achieved a 94% (95% CI 89%-97%) detection rate on the English radiology reports dataset (MIMIC-III) without retraining, suggesting that the core optimization framework is transferable across languages, mitigating the single-language limitation. On the contrary, international mainstream models such as the GPT-4 series exhibited only moderate performance. This disparity underscores the critical impact of linguistic and domain-specific adaptation. Chinese radiology reports present features including word segmentation ambiguity, prevalent terminology abbreviations (eg, “Ca” for cancer), variable omission of quantifiers, and context-dependent semantic expressions. These unique challenges demand a deeply localized LLM solution. DeepSeek-R1, pretrained on high-quality Chinese corpora rich in medical literature, explicitly embeds domain-specific knowledge through adaptive training [21]. This design stands in contrast to GPT-4, whose training data is predominantly English and lacks publicly disclosed optimization for Chinese medical text [22]. The present study highlights that integrating domain-specific knowledge and prompt engineering can unlock robust error detection for non-English clinical narratives while remaining transferable to English data.

A significant advancement of this study was the use of a real-world dataset, as opposed to artificially constructed or synthetic data prevalent in prior research. Gertz et al [11] intentionally inserted 150 errors from 5 common error categories into 100 reports and found that the performance of GPT-4 was comparable to that of radiologists. Sun et al [4] highlighted the potential of fine-tuned LLMs to enhance error detection using LLM-generated synthetic datasets. Although such reports offer better control and avoid privacy concerns and the bias present in real-world data, they may generate new biases, such as overfitting and low generalizability [2]. There may be poor performance in real life if the generated data does not represent a few common scenarios. Other limitations include the low transparency of generated datasets, the risk of perpetuating human-generated bias, and the difficulty of validating them [23]. It should be pointed out that we used artificially inserted errors as the external validation set, which may also not fully represent the complexity and subtlety of real-world English radiology report errors. To bridge these gaps, our primary Chinese dataset captures 1551 naturally occurring errors extracted from the routine radiology workflow, ensuring that the evaluated performance is representative of genuine clinical scenarios, enhancing the practical applicability of our findings.

Compared with human experts, enhanced LLMs, particularly DeepSeek-R1, demonstrated strong error-detection capabilities in radiology reports, surpassing the performance of human readers and matching radiologists on certain tasks. In addition, DeepSeek-R1 performed well in error-type classification and maintained stable detection across imaging modalities with a low FPR. These findings align with prior studies [20] and support their potential as assistive tools for quality assurance, especially amid growing clinical workloads. To better apply LLMs to clinical practice, in addition to considering detection rate, time consumption should also be taken into account. We found that DeepSeek-R1 achieved a 4% increase in detection rate compared to Claude-3.5-Sonnet but required 6 times the processing time. DeepSeek-R1 has 671 billion parameters, 61 layers of transformers embedded with multihead latent attention and mixture-of-experts layers, generating thousands to tens of thousands of internal reasoning tokens before producing the final answer, resulting in significantly longer output times [21]. However, Claude-3.5-Sonnet prioritizes speed through architectural simplicity and implicit inference. On the contrary, when this study was conducted, DeepSeek-R1 was priced at US $0.55 per million input tokens, which was significantly lower than Claude-3.5-Sonnet, priced at US $3.00 per million input tokens. Thus, low-cost DeepSeek-R1 may be more suitable for real-world implementation decisions, such as offline quality control rather than real-time assistance. In addition, previous studies have shown that LLMs achieved a lower mean correction cost per report than the most cost-efficient radiologist [11]. Future efforts should focus on optimizing the balance between accuracy and efficiency through model compression and fine-tuning.

Notably, DeepSeek-R1 offers transparency and can be deployed locally within institutional information technology environments at substantially lower costs than proprietary models [21,24]. For local LLM deployment, there are still many challenges that need to be addressed. The first challenge is hardware constraints (eg, graphical processing unit [GPU] memory, input/output [I/O] bottlenecks) and integration hurdles (eg, heterogeneous picture archiving and communication system [PACS] interfaces). We can address these through elastic compute scaling, model quantization, domestic hardware alternatives, tiered storage, containerized deployment, and standardized HL7/FHIR (Health Level 7 International/Fast Health Care Interoperability Resources) application programming interfaces (APIs). For data security, a “data never leaves the hospital” architecture can be enforced using on-premises GPU inference, SM4/AES-256 encryption, automatic deidentification, retrieval-augmented generation for continuous knowledge updates, and confidence-based filtering to mitigate hallucinations. On the personnel and organizational front, nonintrusive user interfaces (inline highlighting, sidebar summaries) and explainable outputs (eg, error-type justifications) foster radiologist trust. The LLM only suggests corrections and deliberately disables auto-correction; the radiologist retains full authority to accept, modify, or reject each suggestion. Figure S1 in Multimedia Appendix 1 illustrates the proposed clinical workflow: (1) radiologist completes report; (2) LLM runs in background (local deployment); (3) if no error detected → report finalized; (4) if error detected → LLM highlights error and suggests correction; (5) radiologist reviews and then accepts, rejects, or modifies; and (6) final report signed. By integrating via a local API, it eliminates the need to transfer protected data or retrain models, ensuring both high performance and privacy compliance, which can be integrated into radiology reporting systems and effectively handle real-world medical data in clinical practice (Figure S2 in Multimedia Appendix 1).

A critical consideration for the clinical deployment of any AI-assisted error detection tool is its FPR, as excessive false alarms can undermine trust, increase radiologist workload, and lead to “alert fatigue” [20]. We found that DeepSeek-R1 achieved an FPR of 3% on internal validation, which was comparable to radiologists and significantly lower than nonradiologists and nonphysicians. This indicates that enhanced LLMs can maintain high specificity and minimize unnecessary interruptions for clinicians. However, certain errors, such as minor diagnostic omissions and retained normal templates, were still missed (see Table S3 in Multimedia Appendix 1). This necessitates that the final adjudication remain with the radiologist, reinforcing a human-in-the-loop paradigm where the AI acts as a sensitive assistant, not an autonomous arbiter.

Explainability is critical in health care AI to foster trust and support clinical decision-making. Prior studies have largely focused on error detection without evaluating LLMs’ ability to correct errors, which limits their explainability [11,12]. Kim et al [25] demonstrated that GPT-4 could effectively revise head CT reports. Similarly, we found that DeepSeek-R1 achieved 95% accuracy in proposed corrections. Our LLMs can provide an analysis process, reasons for correction, and suggestions for correction to increase clinical applicability. Radiologists can understand the entire correction process in order to make better decisions (accept, modify, or reject). We conducted a comprehensive qualitative analysis of all corrections and found that the LLMs were reasonable for most types of corrections, including simple typographic or grammatical, intermediate terminology or lateralization, and complex semantic rephrasing. However, there are still a small number of corrections that cannot be accepted, such as the use of professional terminology, general conclusions, specific writing habits, and so on. Representative examples of successful corrections and failed corrections were added in Table S3 in Multimedia Appendix 1. The accuracy of correction can be continuously improved by providing real-time feedback to the LLMs on the modifications and rejections provided by radiologists, in order to achieve personalized service levels. These results indicate that LLMs can support an interactive learning environment for residents by identifying common errors and providing real-time feedback, thereby promoting continuous education [11].

This study has several limitations. First, its retrospective design may introduce selection bias. Second, error annotation was restricted to the text level; detecting imaging-report discordance still requires visual input. Future integration of vision-language models could enable end-to-end quality assurance. Third, LLM hallucination remains a key concern, undermining clinical reliability and necessitating expert verification. Although illusions were mitigated to some extent via lower temperature and prompt constraints, fundamental solutions will require greater improvements in model architecture, knowledge injection, and training data control. Fourth, it should be noted that the external validation set (MIMIC-III) used artificially inserted errors, whereas our primary Chinese dataset contained real clinical errors. This difference may affect the generalizability of the cross-language comparison. Although the high detection rates on MIMIC-III (94% for DeepSeek-R1) suggest that the prompt engineering framework is transferable across languages, the artificial nature of these errors may not fully represent the complexity and subtlety of real-world English radiology report errors. Therefore, our cross-language findings could be interpreted as preliminary evidence of generalizability. Future work should validate on real English clinical errors to confirm these observations. Finally, prompt optimization enhances performance but poses standardization challenges and may yield variable effects across models.

Based on a large-scale and real-world clinical corpus, this study demonstrates that enhanced LLMs, particularly DeepSeek-R1, can effectively detect and correct errors in Chinese radiology reports. The findings validate the use of multilingual, clinically derived data and support the local deployment of open-source LLMs as transparent, cost-effective, and privacy-preserving tools within radiology workflows, thereby facilitating their transition toward clinical decision support. While our findings support the potential clinical use of LLMs for error detection, prospective multicenter validation and workflow-based safety assessment are required before routine implementation. In the future, we will conduct multicenter validation to assess generalizability and explore multimodal LLMs for verifying image-text consistency, advancing toward AI-driven quality control in practice.