This systematic review was conducted in accordance with the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) 2020 statement (Checklist 1) [40]. Literature search reporting followed the PRISMA-S (Preferred Reporting Items for Systematic Reviews and Meta-Analyses Literature Search Extension) [41], and narrative synthesis was reported with attention to the Synthesis Without Meta-Analysis (SWiM) guidance [42]. The review question, primary outcomes, and subgroup structure were specified a priori, and the review was registered with PROSPERO (International Prospective Register of Systematic Reviews) 2026 (CRD420261302844) before formal screening began. The prespecified review protocol and extraction backbone are available from the corresponding author upon reasonable request. The PRISMA-S checklist was completed item by item in the Checklist 1, and when a PRISMA-S item was not part of the search methodology or was not applicable, it was explicitly marked and justified.

Eligible studies evaluated generative AI systems based on LLMs for generating an imaging report, report draft, or diagnostic impression from medical imaging, pathology whole-slide imaging, endoscopy, or multimodal, clinically relevant inputs. For eligibility, an intervention was considered eligible if an LLM, multimodal LLM, vision-language foundation model, or a language model component contributed directly to clinical report text generation, diagnostic impression generation, or clinical report drafting. Image classifiers, segmentation tools, retrieval systems, and image captioning systems were not eligible unless a language model or foundation model contributed directly to report generation.

Prespecified task types were report generation from images, generation of impressions from findings, and report drafting. Human oversight was categorized as autonomous generation, human-supervised drafting, or report editing or structuring support. Autonomous generation referred to systems that generated reports or report outputs from images or clinical inputs before human evaluation. Human-supervised drafting referred to workflows in which clinicians actively used, or iteratively collaborated on, AI-generated text during drafting. Report editing or structuring support referred to settings in which AI was applied after source report text, findings, or draft content was already available. We included randomized and nonrandomized comparative studies, reader studies, prospective workflow studies, retrospective validation studies, and single-arm technical validation studies if they used an explicit expert or clinical reference standard. Conference abstracts, reviews, editorials, letters without original data, dissertations, protocols, and preprints were excluded. Because preprints were excluded, this review was designed to evaluate peer-reviewed clinical evidence rather than the absolute frontier of technical model capability.

Effectiveness outcomes were expert acceptance rate and blinded expert preference. Safety outcomes were clinically significant error rate, omission error rate, and commission error rate. Workflow burden outcomes were reporting time, number of corrections, edit distance, and editing burden after generation. Omission errors were defined as missed clinically relevant findings, whereas commission errors were defined as unsupported or hallucinated findings introduced into the generated report. Exploratory outcomes included discrepancy rates between findings and impressions, and text similarity or language quality metrics, such as BLEU, ROUGE, CIDEr, and BERTScore.

We searched PubMed/MEDLINE, Embase, Web of Science Core Collection, Scopus, and the Cochrane Library from January 1, 2016, through May 15, 2026. Europe PMC, OpenAlex, and reference lists of eligible studies and relevant reviews were used as supplemental sources through May 17, 2026, to identify additional records. Study registries were not used as bibliographic search sources because the eligibility criteria targeted completed peer-reviewed studies of medical report generation; this nonuse is reported explicitly in the PRISMA-S checklist. For studies that reported clinically important workflow or safety outcomes in nonconvertible formats, corresponding authors were contacted to request additional analyzable summary statistics or case-level denominator clarification.

The database strategy combined controlled vocabulary and free-text terms related to LLMs, multimodal LLMs, vision-language models, foundation models, and generative AI; report generation, report drafting, diagnostic impression generation, and structured reporting; and imaging domains including radiology, CT, MRI, radiography, pathology whole-slide imaging, ultrasound, endoscopy, and other medical image or signal interpretation settings. Exact database-specific search strings, interfaces, limits, dates, and record counts for PubMed/MEDLINE, Embase, Web of Science Core Collection, Scopus, and the Cochrane Library are provided in Multimedia Appendix 1; the completed PRISMA-S item-level reporting checklist is provided in the reporting guideline checklist file. Database searches were first completed on March 18, 2026, and were updated in May 2026 before final analysis. Across the original and updated database searches, 6936 records were identified before deduplication. Records retrieved in both searches were deduplicated by DOI, PMID, or normalized title and were counted once. After removal of 3008 duplicate records and 499 records already represented in the search corpus, 3429 database records remained for title and abstract screening.

Title and abstract screening and full-text eligibility assessment were performed independently by 2 reviewers after deduplication, with disagreements resolved by discussion or adjudication by a third reviewer. No interrater reliability coefficient was generated; therefore, reviewer agreement is described procedurally rather than as a kappa statistic. Full-text exclusions were logged with one dominant reason per report.

Extraction frameworks at the study and outcome levels were used, and key extraction fields were checked in duplicate for studies that contributed clinically important safety or workflow outcomes. Extracted variables included study characteristics, clinical specialty, imaging modality, task type, level of human oversight, study design, validation setting, model characteristics, comparator type, reference standard, outcome definitions, and raw data sufficient for effect-size computation when available.

Risk of bias was assessed according to study design. Comparative and workflow studies were appraised using ROBINS-I (Risk of Bias in Non-Randomized Studies of Interventions) logic [43], with reporting appraisal informed by CLAIM (Checklist for Artificial Intelligence in Medical Imaging) as an adjunct for assessing AI reporting quality [44]. Retrospective technical validation studies that were not well captured by conventional intervention tools were assessed using a prespecified custom AI validation risk-of-bias framework. The core domains of that framework were representativeness of case selection, reference standard quality, blinding of expert assessment, validation independence, data leakage risk, model or prompt transparency, handling of failed or missing generations, and risk from outcome definition or selective reporting.

For the custom framework, moderate concern was assigned when a limitation was present but limited in scope or of limited consequence for clinical interpretation. High concern was assigned when a limitation was important and likely to influence interpretation, such as unclear validation independence, incomplete blinding, limited external validation, or incomplete failure reporting. Serious concern was assigned when a limitation could directly undermine clinical validity, such as likely data leakage, clearly nonrepresentative case selection, materially weak or unclear reference standards, the absence of handling of failed generations when failures could alter outcome interpretation, or strongly selective reporting of clinically important outcomes. The framework was used as a structured, qualitative, domain-based appraisal rather than a formally validated measurement instrument, and no interrater reliability coefficient was generated for this review. Its judgments should, therefore, be interpreted as transparent qualitative risk signals rather than as validated scale scores.

Because no outcomes met the criteria for meta-analysis, certainty was judged using a narrative certainty assessment informed by GRADE (Grading of Recommendations Assessment, Development, and Evaluation) domains rather than rating pooled effect estimates [45]. We summarized these judgments in a GRADE Summary of Findings table format, using outcome-domain study counts from the selected clinically interpretable end point studies, and reported relative effects as not pooled or not estimable when compatible effect structures were unavailable.

Eligibility for quantitative synthesis was assessed at the outcome level according to clinical coherence, task structure, comparator or reference standard design, and availability of analyzable event counts or summary statistics. For outcomes amenable to direct quantitative reconstruction, planned effect measures were risk differences for dichotomous outcomes and mean differences for continuous outcomes, with 95% CIs when derivable from published data. Outcomes reported across reader-case pairs were not treated as independent case-level events when clustering could not be reconstructed.

Quantitative synthesis was prespecified only for outcomes reported by at least 2 studies with compatible definitions, task structures, comparator or reference standard designs, and analyzable data structures. Outcomes that did not meet these criteria were summarized using structured narrative synthesis. When individual studies provided sufficient data, targeted quantitative reconstruction was performed to support interpretation without pooling. Narrative synthesis was organized by modality, task type, outcome domain, and level of human oversight. Formal subgroup, sensitivity, heterogeneity, and reporting bias analyses were reserved for poolable synthesis sets.

The protocol anticipated quantitative synthesis when clinically and methodologically compatible synthesis sets were available. Supplemental citation discovery and reference screening were added to improve retrieval completeness. These procedures did not change the review question, primary outcomes, eligibility framework, or planned subgroup structure. When outcomes did not meet pooling criteria, synthesis followed the prespecified nonpooled approach of structured narrative synthesis with targeted quantitative reconstruction of individual studies where defensible.

The full study selection process is shown in Figure 1. Across the original and updated database searches, 6936 records were identified before deduplication. After removal of 3008 duplicate records and 499 records already represented in the search corpus, 3429 database records proceeded to title and abstract screening. Public supplemental screening and citation-discovery screening contributed 1 additional eligible study, whereas backward reference screening did not identify additional reports for full-text assessment. Overall, 101 studies met the inclusion criteria for this systematic review. Study-level records are provided in Supplementary Table S1 in Multimedia Appendix 2, and full-text exclusions are listed in Supplementary Table S2 in Multimedia Appendix 2.

The included literature was broad in technical scope but remained concentrated in a limited set of clinical scenarios (Table 1). Of the 101 included studies, 68 (67.3%) were retrospective or technical validation studies, 28 (27.7%) were nonrandomized comparative, reader, or workflow studies, and 5 (5.0%) were prospective studies or real-world workflow or validation studies. Chest x-ray remained the largest modality group (36, 35.6%), followed by mixed-modality settings (20, 19.8%), CT, CT angiography, or positron emission tomography/CT (11, 10.9%), MRI (10, 9.9%), ultrasound (6, 5.9%), endoscopy (5, 5.0%), pathology reporting settings (5, 5.0%), other x-ray or radiography (5, 5.0%), ophthalmic imaging (2, 2.0%), and electrocardiogram (1, 1.0%). By task type, 61 (60.4%) studies evaluated report generation from images, 29 (28.7%) evaluated report drafting or structured report generation, and 11 (10.9%) evaluated impression or diagnostic conclusion generation from findings or reports. By human oversight, 66 (65.3%) studies were classified as autonomous generation, 15 (14.9%) as human-supervised drafting, and 20 (19.8%) as report editing or structuring support. The relationship among modality, task type, and human oversight is shown in Figure 2, and the rapid expansion of the literature after 2023 is summarized in Figure 3A.

Outcome reporting was imbalanced. Benchmark metrics such as BLEU, ROUGE, CIDEr, and BERTScore were common, especially in chest x-ray and technical validation studies, whereas clinically important outcomes, including significant errors, omissions, commissions, expert acceptance, blinded preference, reporting time, and edit burden, were inconsistently reported and used heterogeneous denominators. Thirty-six studies reported at least one clinically relevant human evaluation, safety, workflow, report quality, acceptability, or time outcome, but these outcomes differed by task, comparator, denominator, and reporting format. Benchmark-heavy chest x-ray studies were represented among the included studies [46-50]. Standardized evaluation signal reporting is summarized in Figure 3B. The subset of studies with clinically interpretable effectiveness, safety, and workflow-burden end points is summarized in Table 2.

The risk-of-bias assessment framework is provided in Supplementary Table S3A in Multimedia Appendix 2. Overall risk of bias was moderate in 15 studies, high in 72, and serious in 14; no study was at low risk (Table 3). Although the evidence base included multicenter, external validation, and prospective validation studies, most studies were vulnerable to retrospective selection, incomplete external validation, nonrandomized workflow or reader comparisons, incomplete failure reporting, limited transparency in prompting or model handling, and selective emphasis on benchmark outcomes. Recurrent custom-framework concerns involved validation independence or leakage risk, absent or unclear blinding, heterogeneous outcome definitions, and poor accounting for failed or unusable reports. Risk-of-bias judgments are summarized in Table 3; complete study-level judgments are provided in Supplementary Table S3B in Multimedia Appendix 2; and domain-level bias patterns are shown in Figure 3C.

Certainty was judged very low across all 4 clinically important outcome domains in the GRADE Summary of Findings table (Table 4). This rating does not mean that all findings were clinically unfavorable; rather, it reflects a high or serious risk of bias, incompatible outcome definitions, indirectness across task types and clinical sources, sparse or nonreconstructable effect data, and concern that clinically unfavorable or failed report-generation outputs may be incompletely reported.

Table 2 summarizes studies with clinically relevant human evaluation, safety, or workflow findings using standardized outcome domains, while key extractable quantitative findings are reported in this section, where they can be interpreted with study-specific context. In a large chest radiograph study of autonomous AI preliminary reports, acceptance was 70.5% (6047/8580) for AI-generated reports versus 73.3% (6288/8580) for radiologist reports, whereas false-positive findings were 11.3% (1527/8580) versus 9.7% (831/8580) and false-negative findings were 18.5% (1584/8580) versus 17.8% (1527/8580) in clustered reader evaluations [51]. These evaluations were repeated reader-report or reader-case ratings rather than independent case-level events. In a clinician collaboration study for chest radiograph report generation, AI-generated reports were equivalent to or preferred over clinician reports in 77.7% (233/300) of one dataset and 56.1% (170/303) of another when judged by at least half of the raters, but clinically significant errors occurred in both AI-generated and clinician-generated reports [52]. Outcome-level extraction details are provided in Supplementary Table S4 in Multimedia Appendix 2.

Other human evaluation studies of autonomous multimodal generation reported expert indistinguishability, acceptance, or quality ratings in 3D brain CT report generation, bronchoscopy report generation, and medical vision–language reporting across mixed devices [55-57]. Related autonomous report generation studies in chest x-ray, MRI, and fine-tuned LLaMA-based pipelines reported technical performance in additional settings [5,6,72].

Workflow outcomes differed across task types. In a brain MRI reader study, AI assistance improved diagnostic performance and reduced reading time from 61 to 53 seconds, with greater reported benefit among junior radiologists [53]. In contrast, in a study of radiological impression drafting with multiple readers, drafts generated by the model required more editing time than the radiologist baseline (18.29 vs 12.20 s) and a greater edit distance (12.32 vs 5.74 words changed) [54]. For this study of impression drafting, arm-specific means and CIs were sufficient to support the cautious quantitative reconstruction of single studies for sensitivity purposes, but no second clinically comparable workflow study was available for pooling (Supplementary Table S5 in Multimedia Appendix 2) [54]. Additional workflow studies in lumbar spine MRI, breast ultrasound, structured positron emission tomography/CT reporting, and practical drafting with ChatGPT reported on workflow integration in heterogeneous settings [73-76].

Additional studies in the evidence base evaluated blinded radiologist assessment, structured report reader experiments, emergency CT impression drafting with assessment of critical omissions, real-world or pilot reporting workflows, prospective endoscopy report validation, and reporting time comparisons with LLM assistance (Table 2). Reported findings varied by task: some studies reported improved reporting efficiency, structured report usability, or expert acceptability, whereas others reported persistent omissions, hallucinations, lower quality ratings from specialists compared with human reports, or workflow benefits limited to narrow, standardized settings. The characteristics of studies added to the final evidence set are summarized in Supplementary Table S1 in Multimedia Appendix 2 [58-71,77-130].

No prespecified effectiveness, safety, or workflow burden outcome had at least 2 studies with directly comparable definitions, coherent designs, and analyzable effect structures suitable for meta-analysis. The expanded evidence base broadened clinical contexts but also added heterogeneity in inputs, outputs, comparators, human oversight, and denominator definitions. The strongest quantitative workflow signal came from the single impression-drafting study in Table 2 and Supplementary Table S5 in Multimedia Appendix 2. Safety and expert evaluation outcomes were usually reported as clustered reader-level counts, threshold preferences, paired overlap summaries, qualitative ratings, or pilot metrics that could not be pooled. These reporting barriers were summarized in Supplementary Table S6 in Multimedia Appendix 2 and structured the narrative synthesis.

No additional unpublished outcome data were available by May 19, 2026; therefore, all quantitative reconstruction and narrative synthesis relied on publicly reported results.

This systematic review assessed the effectiveness (expert acceptance and blinded preference), safety (clinically significant errors, omission errors, and commission errors), and workflow burden (reporting time, corrections, edit distance, and editing burden) of LLM-based medical report generation. Relative to these objectives, the principal finding is that the field is active and clinically diverse, but the reported evidence remains too heterogeneous to support stable conclusions about overall effectiveness, safety, or workflow benefit. Effectiveness outcomes suggested possible assistive value in selected settings [51,52], but these outcomes were not consistently paired with case-level safety end points. Safety and workflow burden outcomes were reported with incompatible denominators, task structures, and comparison designs [53,54].

The findings, therefore, support a cautious, task-specific interpretation rather than a general claim that LLM-based report generation is effective or safe. Some supervised workflows appeared acceptable to experts or were useful for selected reporting tasks, but current evidence does not justify autonomous replacement of clinicians [20,23,131]. Benefit and risk differed by task and workflow because impression generation from findings, human-supervised drafting, report editing or structuring, and autonomous image-to-report generation involve different clinical inputs, opportunities for human correction, and failure modes. These task categories should not be treated as interchangeable when interpreting acceptance, reporting errors, workflow burden, or implementation readiness.

Prior reviews and commentaries have described the rapid emergence of AI report generation, structured reporting, prompt engineering, and foundation-model applications in imaging and endoscopy [24-28]. Recent systematic and scoping reviews have further summarized automated radiology report generation, structured reporting approaches, and deep learning trends [29-31], as well as LLM use in radiology reports, readability, simplification, and broader radiology adoption trajectories [32-34]. These publications establish that the technical literature is active, but leave a practical clinical question unresolved: whether published studies report clinically interpretable effectiveness, safety, and workflow outcomes in a way that can guide supervised implementation. This review differs from those prior reviews by prioritizing outcome definitions, the level of human oversight, and synthesis feasibility rather than technical capability alone. This emphasis allowed us to map not only what models can generate but also how reported outcomes relate to clinical usefulness.

The effectiveness findings need to be interpreted according to the outcomes that were actually reported. Expert acceptance, blinded preference, and report-quality judgments were the most common effectiveness signals, and several studies suggested that generated reports or AI-assisted reports could be acceptable, equivalent, or preferred in selected reader-evaluation settings [51,52]. However, acceptance and preference are not the same as clinical effectiveness unless they are accompanied by evidence that final reports are accurate, clinically complete, and not more burdensome to verify [20,23]. In this review, positive effectiveness signals were often limited by single-study designs, clustered reader ratings, heterogeneous rating scales, and incomplete linkage to omissions, commissions, or final clinician-edited report quality. Therefore, the evidence supports a narrower conclusion: LLM-generated or LLM-assisted report text may be useful in some supervised contexts, but published outcomes do not yet establish a generalizable effectiveness advantage across medical reporting workflows.

The safety and workflow outcomes further explain why broad effectiveness claims would be premature. In chest x-ray and clinician-collaboration settings, expert acceptance or preference could coexist with false-negative findings or clinically significant errors, showing that acceptable language does not necessarily imply safe reporting [51,52]. Workflow findings were also mixed: a brain MRI study reported shorter reading time with AI assistance, whereas an impression-drafting study reported longer editing time and greater edit distance than the radiologist baseline [53,54]. These findings suggest that workflow benefit depends on where the system enters the reporting process, whether clinicians start from images, findings, or draft text, and how much verification or correction the AI output requires.

The nonpoolability of the evidence base is itself an important synthesis finding. Meta-analysis was inappropriate not because publication volume was small, but because the evidence structure was incompatible across studies. Clustered reader-level counts, threshold preference summaries, paired workflow results without reconstructible dispersion, heterogeneous quality scales, pilot outcomes, and unclear denominators prevented valid pooling. This pattern aligns with SWiM principles: when effect estimates cannot be validly combined, transparent, structured narrative synthesis is preferable to forced quantitative aggregation [42]. The expanded search increased clinical breadth but did not create a coherent synthesis set for any primary or key secondary outcome, indicating a reporting-standardization problem that limits cumulative clinical inference.

The dominance of chest x-ray studies also needs careful interpretation. Public corpora such as MIMIC-CXR have accelerated model development and benchmarking [15,16], but repeated use of the same or related datasets can increase the risk of overlapping test sets, model and data leakage, and overly optimistic generalizability. This concern is especially relevant when training, validation, and external testing boundaries are not reported clearly. Endoscopy, pathology whole-slide imaging, CT, MRI, and ultrasound reporting involve different information densities, reference standard structures, and workflow constraints; consequently, evidence from templated chest radiography should not be generalized uncritically to more complex reporting domains.

The safety findings also require interpretation at the level of the clinical task. Human preference or acceptance of generated text should not be interpreted as equivalent to safety unless omissions, commissions, clinically significant discrepancies, and failed generations are measured explicitly [19,21,23]. A fluent report can still omit an important abnormality, introduce an unsupported finding, or increase the review burden for the clinician who must verify it [21-23]. This distinction is especially important for autonomous image-to-report systems, where errors may arise before a clinician’s review, but it also matters for impression-drafting and report-structuring tools because they can shape the language and emphasis of the final report. Accordingly, implementation decisions should require direct measurement of error types in the specific workflow under consideration.

For clinical practice, the real-world implication is that LLM-based reporting systems should be evaluated as supervised assistive tools, not as unsupervised replacements [23,131]. Implementation decisions should be workflow specific because an AI tool that drafts an impression from existing findings has a different safety profile from an autonomous image-to-report model, and a tool that improves readability or structure does not necessarily reduce diagnostic error. Local validation should measure the task actually being deployed, the final clinician-edited report, and the workload created by reviewing or correcting generated text [20,23,131]. This means evaluating systems in the local modality, report template, patient mix, clinician review process, and information-technology environment in which they would actually be used.

For developers and clinical investigators, these findings point toward a staged evidence pathway. Benchmark metrics may be appropriate for early technical screening, but progression toward clinical use should require prospective or externally validated studies with safety outcomes at the case level and reconstructable workflow statistics [20,23,131]. Evaluation should separate omissions from commissions, report failed or unusable generations, state case-level denominators, and measure reporting time, corrections, edit distance, acceptance, blinded preference, and final report quality in realistic workflows [23,44,131]. These reporting elements would make future evidence more useful for health systems because they connect apparent effectiveness to the safety and labor required to produce a clinically usable final report.

For evidence synthesis, the field needs more consistent reporting before meta-analysis can become informative. Future studies should define whether the unit of analysis is the patient, examination, report, reader-report pair, or generated text segment; specify whether evaluators were blinded; report how unusable generations were handled; and provide enough event counts or summary statistics to reconstruct effects [42,44,131]. They should also report effectiveness, safety, and workflow outcomes together rather than presenting acceptance, errors, and time savings as isolated signals. Without these elements, additional studies may increase publication volume without improving certainty or allowing health systems to judge whether LLM-assisted reporting improves outcomes that matter in practice.

This review has limitations. The evidence base was dominated by retrospective studies; prospective workflow research was sparse; and no main outcome supported a robust pooled estimate across multiple directly comparable studies. The custom AI validation framework was prespecified and transparently domain-based, but it was not a formally validated measurement instrument, and interrater agreement statistics were not generated; accordingly, those judgments should be interpreted as structured qualitative bias signals rather than as validated scale outputs. The narrative GRADE-informed certainty assessment was likewise constrained by the absence of pooled estimates; it used GRADE domains to structure certainty judgments but did not produce certainty ratings around pooled effect sizes [45].

Excluding preprints may underrepresent the newest model capabilities in a fast-moving field, but this choice was appropriate for a clinical review focused on peer-reviewed evidence. Some included and excluded studies may share public datasets or related benchmarks, reducing independence across studies [15,16]. We also could not always determine whether model training data overlapped with evaluation data, especially for studies using public radiology corpora or proprietary foundation models. Finally, structured narrative synthesis provides less statistical compression than meta-analysis but better reflects the current nonpoolable evidence structure [42].

This review provides a clinically oriented synthesis of LLM-based medical report generation across effectiveness, safety, workflow burden, and human oversight. It differs from radiology-specific, readability-oriented, and benchmark-focused reviews by asking whether peer-reviewed evidence can support clinical judgments about expert acceptance, blinded preference, clinically significant errors, omissions, commissions, reporting time, corrections, edit distance, and editing burden. Its main contribution is to show that the barrier is no longer simply whether models can produce plausible reports; it is whether studies report acceptance, preference, omissions, commissions, failed generations, time, corrections, edit distance, and human oversight well enough to guide practice. The broader real-world implication is that adoption should remain assistive, clinician-supervised, locally validated, and specific to the intended reporting workflow until stronger prospective safety and workflow evidence is available [20,23,131]. In practical terms, LLM-based reporting should be viewed as a candidate workflow aid whose value depends on the clinical task, validation setting, and human review process, not as a general replacement for expert report generation. The next stage of research should therefore move from benchmark-centered demonstrations toward standardized, case-level effectiveness, safety, and workflow burden evaluations that can support both local implementation decisions and future quantitative synthesis.