As an initial step, multiple open-weight instruction-following LLMs were evaluated, served via vLLM [14] and queried using the OpenAI Python package [15], with inference parameters configured per official documentation (Multimedia Appendix 1). The models were selected based on 3 criteria: strong general-purpose generation capabilities, competitive multilingual coverage including German, and the feasibility of deployment on a single NVIDIA A100 (80 GB). Models exceeding 70B parameters were executed in 4-bit AWQ (activation-aware weight quantization) [16], except for the OpenAI models, which were natively released in MXFP4 (microscaling floating point 4) quantization [17]. The evaluated models deliberately span high-capacity state-of-the-art baselines and efficient small- to medium-sized models to quantify quality-efficiency trade-offs under realistic hardware constraints. These are summarized in Table 2.

This study was designed and reported in accordance with the TRIPOD-LLM (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) [18] guideline for studies evaluating LLMs in health care.

MTB patient protocols follow defined section schemes, and structural malformation errors have been shown to have a significantly negative impact on clinical evaluation scores [24]. Two straightforward approaches for enforcing schema-compliant protocol generation were considered and rejected. Section-wise prompting, as used in our preliminary work, generates each section independently and cannot guarantee narrative coherence across the full protocol. Naive prompt-based template filling, where a protocol template is provided in the prompt and the model is instructed to complete it, proved brittle in pilot runs, as it produced altered headers, superfluous sections, and omitted mandatory fields.

Structured generation addresses both limitations by enforcing schema compliance at the token level through constrained decoding. Rather than instructing the model via natural language, constrained decoding, implemented through llguidance [25], mathematically prevents the model from generating any output that violates a predefined format. Separate JSON schemas were defined for protocols with and without therapy recommendation. As fixed structural elements are guaranteed by the schema, clinician review is limited to dynamic content sections only. This minimizes the oversight burden. The final protocols were rendered from the structured model output using a Jinja2 [26] template.

Comparison of zero-shot and one-shot approaches was a key component of the analysis. As the available data precluded fine-tuning, both approaches used prompt-based control, which mirrors the constraints common in hospital settings. Complete prompt templates and schemas are provided in Multimedia Appendix 2.

Regarding approach 1, structured template filling (A1) used zero-shot, schema-driven generation without examples. A system prompt specified persona, rules, and style constraints. The user prompt contained the generation instruction, short field-level instructions injected via the JSON schema, and the MTB protocol as source content.

Regarding approach 2, style-conditioned structured template filling (A2) extended A1 with one-shot, style-conditioned generation, motivated by in-context learning [6]. The same schema and structured decoding were used, with one expert-written example injected per dynamic field, following the field instructions to anchor tone, lexical choices, and level of detail to institutional practices.

Given the limited gold standards and the requirement for German-suitable metrics, automatic evaluation combined reference-based fidelity metrics with reference-free readability and complexity measures. Reference-based metrics were computed for the 410 protocol sections by aligning the generated JSON fields with the corresponding sections in the expert-written protocols. These reference protocols were written using a structured framework grounded in didactic and psychological principles, codeveloped by MTB physicians, psychologists, and the WTZ patient advisory board [13]. Reference-based metrics should therefore be interpreted as measures of stylistic and structural proximity to this reference, not as absolute indicators of lay-language quality.

Lexical fidelity was assessed using ROUGE-1 (Recall-Oriented Understudy for Gisting Evaluation) [27], which measures the word-level overlap between generated and reference texts (0‐1, higher is better).

Semantic fidelity was assessed using BERTScore-F1 (Bidirectional Encoder Representations From Transformers Score) [28], which computes embedding-level similarity to capture paraphrases and conceptual overlap (0‐1, higher is better).

Readability metrics can be applied without gold standards because they rely solely on text features, such as sentence and word length. Readability was assessed using Wiener Sachtextformel version 4 (WSTF4) [29], a formula optimized for German, computed from the MS (percentage of words with 3 or more syllables) and the SL (average sentence length in words):

Scores of 4‐6 indicate very easy texts, 7‐10 well-understood, 10‐12 medium difficulty, and above 12 specialist-level; scores can additionally be interpreted as the approximate school grade level required for comprehension.

Text complexity was assessed using a DistilBERT (Distilled Version of Bidirectional Encoder Representations From Transformers) model fine-tuned on German texts [30]. The model yields scores on a 1‐7 scale, with lower values indicating simpler texts.

The differences in verbosity were quantified as the relative length differences between the generated and gold standard sections in words. A verbosity score of 1−|Δlength| was used, where higher values indicate lower deviation. This score is a proximity measure with a range of (−∞, 1], and no clipping or normalization was applied. All verbosity results are reported as medians, which remain robust against the extreme values caused by overgeneration.

Automated metrics provide a scalable proxy for linguistic fluency but are notoriously brittle in clinical contexts due to their inability to capture medical safety implications or the specific error types inherent in LLM-generated outputs.

Rigorous evaluation of patient protocols requires a taxonomy that covers factual correctness, communicative adequacy, and patient safety. Established frameworks such as the Physician Documentation Quality Instrument-9 [31] or SummEval [32] have been proven useful, yet both assign scores to quality criteria rather than pinpointing specific errors, a level of detail that more closely reflects the correction work clinicians actually perform. Error annotation frameworks proposed for summarization [33], translation [34], and clinical documentation [10] offer finer-grained analysis but treat any addition beyond the source document as an error or hallucination. This assumption represents a fundamental mismatch for patient protocol generation, where explanatory additions are necessary to achieve health literacy alignment. Penalizing such elaborations would systematically underestimate quality and mischaracterize safe model behavior.

Therefore, a specific taxonomy was developed. The taxonomy proposed here classifies each identified issue along 3 dimensions: 4 error types (factual, omission, noise, and language), 2 contextual scopes (internal and external), and 3 severity levels (minor, major, and critical). Reports that contain at least one critical error are rated as invalid and not acceptable for clinical use.

The scope dimension reflects the dual information sources of patient protocols: the factual content of the MTB protocol (internal) and the general medical background knowledge required to make the content comprehensible to lay readers (external). This distinction prevents the misclassification of clinically necessary explanatory additions as hallucinations or factual errors.

The four error types are defined as follows: (1) factual errors occur when the text contradicts either the source document (internal), such as incorrect diagnoses or therapy recommendations, or established medical knowledge (external), such as a misdefined biomarker; (2) omission errors occur when relevant information is absent. Internal omissions cover patient-specific details from the source document, and external omissions cover missing background explanations necessary for lay understanding; (3) noise errors occur when a text contains unnecessary or misleading content. Internal noise comprises redundant details from the source document, and external noise refers to unrelated medical information that dilutes key messages; and (4) language errors are linguistic or stylistic issues (eg, grammar, syntax, or phrasing) that do not affect factual content but may reduce readability, professionalism, or patient trust.

Severity reflects the potential impact on patient understanding and safety. Minor errors involve terminological or stylistic inaccuracies with no safety implications. Major errors substantially impair lay comprehension or distort key medical facts. The patient may misunderstand content, yet no direct harm or dangerous consequence is likely to result. Critical errors carry direct patient safety implications and risk harm through misinformation, such as incorrect therapy instructions or the omission of essential safety warnings. Annotators must assign the highest applicable severity level.

Following best practices from machine translation human evaluation research, only domain experts participated in the manual evaluation, given the interpretive complexity of clinical content. Seven board-certified medical oncologists or hematologists from the WTZ served as annotators, with direct experience in complex MTB cases constituting relevant domain expertise.

A stratified random sample of 25 MTB cases was drawn from the parallel corpus (20 with therapy recommendations and 5 without therapy recommendations). Priority was given to cases with therapy recommendations to ensure a focus on high-complexity reports with more dynamic fields and greater error risk. Consequently, expert-level characterization of protocols without therapy recommendations is more limited. To avoid distortion, there was no “human-in-the-loop” before the expert evaluation. Using the top-performing model from the automatic evaluation, paired A1 and A2 outputs yielded 50 unique protocols. Restricting expert evaluation to a single model was a deliberate choice, as including multiple models was not feasible given the annotation workload, and the primary goal was to compare generation approaches rather than models. Each protocol was independently rated by 2 clinicians, for a total of 100 ratings. As no consensus discussion or Delphi reconciliation was performed, interannotator agreement was quantified post hoc using Gwet agreement coefficient 1 (AC1), as described in the Statistical Analysis section.

A between-subject blinded design preserved ecological validity: each annotator saw each MTB protocol only once, and the generation conditions (A1 vs A2) were undisclosed. This mirrors anticipated routine deployment conditions, where each clinician reviews a given case only once and thereby avoids potential fatigue effects.

Clinical usability was assessed along the 3 ISO 9241‐11 usability dimensions of effectiveness, efficiency, and satisfaction using 3 complementary instruments.

Together, these 3 instruments capture distinct and decision-relevant aspects of usability: error annotation reflects objective output quality, PPEE proxies operational integration burden, and NPS captures broader clinical acceptability.

A dedicated web interface built using Streamlit [39] supported expert evaluation. Hosted within the hospital network and accessible via institutional credentials, it comprised 3 components: a project overview describing this study and error taxonomy, the core evaluation interface, and a visual decision tree guiding annotators through taxonomy category assignment. The evaluation interface used a dual-pane layout. The left panel displayed the original MTB protocol; the right panel presented the generated patient protocol divided into dynamic sections. Annotators first read the full protocol and then assessed each dynamic section for errors. Multiple errors could be recorded per section with optional free-text comments. After completing section-level annotation, annotators rated the PPEE and NPS for the full document. All annotations were saved as JSON files for subsequent analysis.

Annotators took part in a training procedure that included an online session, 2 written reference documents, and 5 practice protocols not part of the final evaluation set. During the online session, this study’s team walked through the taxonomy and the interface and answered annotators’ questions directly. One reference document explained the error taxonomy with definitions and worked examples. The other described the evaluation interface. A decision tree for step-by-step category assignment was embedded in both the interface and the taxonomy document and was available throughout annotation.

For each system (defined as LLM×prompting approach) and each metric, the results are summarized as the median with a 95% percentile bootstrap CI (5000 resamples, stratified by document).

System-level comparisons used pairwise tournament analysis to evaluate all system combinations head-to-head across metrics. For each pair, the proportion of instances in which system i outperformed system j was computed. The net win percentages (wins minus losses) yield an interpretable scale-independent performance ranking.

The primary inferential analysis compared the 2 prompting approaches (A2 vs A1) within each LLM using the Wilcoxon signed-rank test [40] on document-level paired differences (A2−A1). The effect size was quantified using the matched-pairs rank-biserial correlation r, where r=1 indicates perfect superiority of A2 and r=−1 indicates perfect inferiority. Benjamini-Hochberg correction [41] controlled the false discovery rate. All raw P values (8 LLMs×5 metrics) were pooled into one correction family per analysis domain. Adjusted P values below .05 are considered statistically significant.

All analyses were conducted across 3 prespecified subsets (overall, with therapy recommendation, and without therapy recommendation) to account for potential heterogeneity in protocol structure and vocabulary between document types.

Analyses were implemented in Python (version 3.12.11), using NumPy, pandas, SciPy, and statsmodels.

Five effectiveness outcomes were captured at the document level: error count, severity-weighted error score (minor=1, major=2, and critical=3), and errors stratified by type, scope, and severity. Spearman rank-order correlation (ρ) quantified the association between the severity-weighted error score and expert ratings (NPS and PPEE). NPS was additionally analyzed using the standard 3-category classification (detractors: 0‐6, passives: 7‐8, and promoters: 9‐10).

For each approach and outcome, the results are reported as the median with a 95% percentile bootstrap CI (5000 resamples). Before the A1 vs A2 comparisons, error counts were averaged across the 2 raters to obtain a single case-level estimate.

The highly zero-inflated distribution of paired differences (eg, 17/25, 70%, identical outputs in the noise category) made the Wilcoxon signed-rank test inappropriate. Instead, an exact binomial sign test [42] on nontied pairs assessed pairwise approach preference, with Cohen g reported as the effect size measure. Benjamini-Hochberg correction controlled the false discovery rate across all outcomes within each analysis domain. Adjusted P values below .05 are considered statistically significant.

Interannotator agreement on error presence was evaluated using Gwet AC1 [43]. This metric was deliberately chosen to counter the highly imbalanced nature of the evaluation data, where the absence of errors constituted the overwhelming majority of the observations. In such skewed distributions, metrics such as Krippendorff α suffer from the prevalence paradox, severely underestimating reliability. All coefficients included bootstrapped 95% CIs (5000 resamples).

Analyses were implemented in Python (version 3.12.11), using NumPy, SciPy, pandas, and the irrCAC library.

In 16% (230/1420) of annotated paragraphs, experts identified a total of 253 errors, which resulted in a median of 2.0 (IQR 1.5‐3.5) errors per protocol. By approach, A1 yielded a median of 2.0 (IQR 1.5‐2.5) errors per protocol and A2 a median of 2.5 (IQR 1.5‐3.5) errors per protocol. Median paired differences were Δ=0.0 (95% CI −0.5 to 2.0), with no significant differences observed in error counts between A1 and A2.

The proportion of protocols with ≥1 critical error was 20% (n=5) for A1 and 40% (n=10) for A2. The median of critical errors per protocol was 0.0 (IQR 0.0‐0.5) for A1 and 0.5 (IQR 0.0‐1.5) for A2. Major errors occurred at a median of 0.5 (IQR 0.0‐0.5) per protocol for A1 vs 1.0 (IQR 0.0‐1.5) for A2, while minor errors occurred at 1.0 (IQR 0.5‐1.5) for both A1 and A2. Median paired differences in error severity between approaches were Δ=0.0 (95% CI 0.0 to 1.0) for critical errors, Δ=0.5 (95% CI 0.0 to 1.0) for major errors, and Δ=−0.5 (95% CI −1.0 to 0.5) for minor errors.

Critical errors fell into 2 main categories. The largest group, with 24 annotations across 12 protocols, involved misjudgment of clinical relevance. This included providing explanations for therapeutically irrelevant alterations, omitting secondary options such as genetic counseling or trial referrals, and incorrectly characterizing the evidence base. These errors occurred under both prompting conditions, but they were more frequent under condition A2. The increase in critical errors under A2 was driven by exemplar contamination (18 annotations across 6 protocols). The model carried over content from the one-shot exemplar and manifested 3 subtypes: drug or therapy carryover (a drug featured in the exemplar was recommended for a patient for whom it was not indicated), approval-frame carryover (the exemplar’s off-label framing caused the model to misrepresent the approval status of guideline-recommended therapies), and trial-name carryover (clinical trials cited in the exemplar were inserted in place of the patient’s actual evidence base). Multimedia Appendix 4 contains a table listing all critical errors, their types, the sections of the protocols in which they occurred, and descriptions of the deidentified content.

While language was the prominent error type for A1 (40/108, 37%), factual errors were the most prominent type in A2 (69/145, 47.6%). The median of factual errors per protocol was 0.5 (IQR 0.0‐1.0) for A1 and 1.0 (IQR 0.5‐2.0) for A2. Omission errors occurred at a median of 0.5 (IQR 0.0‐1.0) per protocol for both A1 and A2, while noise errors occurred at 0.0 (IQR 0.0‐0.5) for A1 and 0.0 (IQR 0.0‐0.0) for A2. Language errors had a median of 0.5 (IQR 0.5‐1.0) per protocol for A1 and 0.5 (IQR 0.0‐1.0) for A2.

However, median paired differences showed no notable differences in error types between A1 and A2 (factual ∆=0.0, 95% CI 0.0 to 1.5, omission ∆=0.0, 95% CI 0.0 to 0.5, and noise ∆=0.0, 95% CI 0.0 to 0.0) except for language with ∆=−0.5 (95% CI −1.0 to 0.0) while not significant.

Figure 5 shows the distributions of the severity-weighted error scores per protocol. It shows that for factual errors the median paired difference after severity weighting changed to ∆=0.5 (95% CI −0.5 to 4.0). For the other types, there were no notable changes when severity weighting was applied, except for the minimally increased CIs for omission and language.

Looking at the scope of errors (Figure 6), internal errors dominated under both approaches, constituting 88% (60/68) under A1 and 63% (71/112) under A2. The median internal errors per protocol were 1.0 (IQR 0.5‐1.5) for A1 and 1.0 (IQR 0.5‐2.5) for A2. External errors were rare under A1 (12% of all errors; median 0.0, IQR 0.0‐0.0) but increased markedly under A2 (37% of all errors; median 0.5, IQR 0.0‐1.5).

Overall, no significant differences were detected by the exact binomial test in the error type, severity, scope, or severity-weighted error scores between A1 and A2.

The interannotator agreement analysis yielded a substantial aggregate agreement (Gwet AC1=0.72, 95% CI 0.67 to 0.76) for all errors regardless of type, severity, and scope. Almost perfect agreement [44] was reached on critical errors (Gwet AC1=0.94, 95% CI 0.92 to 0.96), which supports the validity of the critical error rate as the safety-relevant outcome of this study. Detailed reliability statistics, stratified by error type, severity, and scope, are provided in Multimedia Appendix 3.

Over all ratings, the PPEE (Figure 7A,C) had a median of 2.0 (95% CI 2.0 to 3.0; IQR 2.0‐3.0). A total of 52% (52/100) of ratings indicated very low to low effort, 24% (24/100) medium, and 24% (24/100) high to very high. For A1, the median was 2.5 (95% CI 2.0 to 3.0; IQR 2.0‐3.0), and for A2, the median was 3.0 (95% CI 2.5 to 3.5; IQR 2.0‐3.5). The median paired difference was Δ=0.5 (95% CI −0.5 to 1.0) and not significant. There was a moderate and significant (P<.001) correlation between PPEE and severity-weighted error score (ρ=0.55).

Across all 100 individual ratings, the NPS (Figure 7B,D) had a median of 7.0 (95% CI 6.0 to 8.0; IQR 5‐9). The category distribution classified 29% (29/100) of ratings as promoters (scores 9‐10), 25% (25/100) as passives (scores 7‐8), and 46% (46/100) as detractors (scores 0‐6). By approach, A1 had a median of 7.0 (95% CI 6.5 to 8.5; IQR 6.0‐9.0) compared to a median of 6.5 for A2 (95% CI 4.0 to 7.0; IQR 3.5‐7.0). The detractor share rose from 40% (20/50) under A1 to 52% (26/50) under A2. Median paired differences were Δ=−1.0 (95% CI −3.0 to 0.5) and nonsignificant. Protocols rated by detractors carried a higher severity-weighted error burden, consistent with the moderate and significant (P<.001) correlation between NPS and severity-weighted error score (ρ=−0.54). Per rater, the detractor share ranged from 14% (2/14) to 86% (12/14), and the median item score from 4.0 (IQR 1.25-5.0) to 10.0 (IQR 10.0-10.0). The detractor predominance was concentrated in 2 raters who together contributed 24 of the 46 detractor ratings. Per-annotator distributions are reported in Multimedia Appendix 3.