This study was approved by the Ethics Committee of the Medical Faculty of the University of Duisburg-Essen (approval 23-11557-BO). Due to the study’s retrospective nature, the requirement of written informed consent was waived by the ethics committee. All data were fully anonymized before being included in the study.

Patients were eligible for inclusion if they had a liver resection between 2010 and 2024, their electronic health record contained at least 1 operative report confirming a hepatic surgical procedure, and at least 1 discharge letter documenting the perioperative course. Across all hepatobiliary service lines—hepatocellular carcinoma (HCC), colorectal liver metastases (CRLM), perihilar cholangiocarcinoma (Klatskin tumors), and intrahepatic cholangiocarcinoma (ICC)—a total of 650 postoperative cases met these criteria and were included after preprocessing (Figure 1).

For each patient, all available discharge and transfer letters were initially retrieved. As part of the expert annotation workflow, the final surgical discharge letter was manually selected for every patient. This expert-selected physician letter was treated as the authoritative postoperative summary and served as the sole input document for model prompting. During data preprocessing, letterhead sections were removed to standardize the narrative input.

Expert labeling was performed by a senior attending surgeon with >10 years of clinical experience. For each case, the annotator was presented with all available discharge, transfer, and investigation letters from the relevant encounter, from which the surgical discharge letter was selected. The Clavien-Dindo grade was assigned exclusively based on the content of the selected letter, without access to additional clinical information such as laboratory values, imaging reports, or intraoperative findings, to ensure strict alignment with the information available to the language models (Figure 2).

The manual annotation process was guided by a designated annotation user interface, which presented concise definitions of all Clavien-Dindo grades. Annotators assigned a single, predefined grade for each case using this structured interface, ensuring standardized input and minimizing formatting variability. Free-text comments could be added to flag ambiguous or borderline cases, facilitating subsequent review and, when necessary, refinement of the consensus labels. The resulting expert labels were used as the reference ground truth for all subsequent analyses.

To assess the reliability of the expert annotation and to establish a human benchmark for model performance evaluation, a stratified random subset of 65 (10% of the full dataset) cases was selected for independent second annotation. Cases were sampled to approximate a balanced grade distribution across all Clavien-Dindo grades. Grades I-V were each represented by 8 cases. An exception was made for grade IVb, which occurred only 6 times in the full dataset and was therefore included in its entirety. Grade 0 was represented by 11 cases, reflecting its proportionally higher prevalence in the overall cohort.

The second annotation was performed by an independent senior surgeon with 9 years of experience in hepatobiliary surgery and established familiarity with the Clavien-Dindo classification system. Both annotators were blinded to each other’s ratings, and the second annotator had no access to the primary expert annotations during the review process. Interrater agreement was quantified using Cohen κ, which accounts for agreement beyond chance. The resulting κ value serves as a human reference benchmark, allowing model-to-expert agreement, also reported as Cohen κ throughout the evaluation, to be interpreted in the context of clinician-to-clinician variability inherent to Clavien-Dindo grading.

For this study, a diverse set of LLMs was selected, encompassing both open-weight and proprietary architectures to assess performance across different model classes and deployment constraints (Table 1). Throughout this manuscript, the term “open-weight” refers to models whose trained parameters are publicly accessible and can be deployed locally, but whose training data, code, or full technical specifications may not be fully disclosed, distinguishing them from strictly “open-source” models, which imply complete transparency across all components. For the open-weight category, we selected 4 current LLMs: Qwen3 (Alibaba Cloud), Llama 3.3 (Meta AI), GPT-OSS (OpenAI), Ministral 3 (Mistral AI). These models were chosen for their strong general-purpose reasoning capabilities, broad community support, and the ability to deploy them locally, ensuring full compliance with institutional data protection regulations. Local deployment also enabled deterministic control over inference settings and resource allocation.

To contextualize performance, we additionally included the 2 state-of-the-art proprietary models: GPT-5.1 and Gemini 3 Pro. These models were accessed via secure API end points (provider: OpenAI API) and selected for their strong benchmark performance in clinical reasoning and information extraction tasks [24-27].

Model interaction was based on a structured prompting framework, which we refined iteratively in consultation with the surgical expert to optimize clarity, task decomposition, and alignment with the Clavien-Dindo classification criteria. All evaluations were conducted in a zero-shot setting, without task-specific fine-tuning or exposure to labeled examples, to assess the intrinsic out-of-the-box capability of each model to infer postoperative complication severity directly from unstructured clinical documentation. This approach was deliberately chosen to establish a reproducible baseline that reflects the models’’ general reasoning ability rather than optimized task-specific performance. The final prompt consisted of 4 sequential components: a concise task description, a definition of the Clavien-Dindo scale, instructions for determining the correct grade, including a required output schema, and the full physician letter for each case. This structure ensured that models received identical contextual and instructional information. Full prompt templates are provided in Supplement S1-S4 in Multimedia Appendix 1.

Prompting was conducted under 2 distinct scenarios (Figure 3). In the first scenario, models were asked to predict the fine-grained Clavien-Dindo grade (0-V). In the second scenario, we derived a binary classification scheme in which the grades were grouped as minor complications (grades 0-IIIa) and major complications (grades IIIb-V). For each model output, we extracted 3 predefined fields, including score or category, reason, and citation, to enable structured downstream analysis. The same prompt was applied to all models without model-specific modifications. Additional context length constraints varied across models, and the Llama-3.3-70B was limited to approximately 6000 tokens, requiring truncation of a few excessively long physician letters. In such cases, appendices without additional clinical content were removed after manual review to retain the document’s clinically relevant core. All further hyperparameters are listed in Table S1 in Multimedia Appendix 1.

Model performance was assessed by comparing the Clavien-Dindo grades predicted by each LLM with the expert-assigned ground truth labels. Evaluation metrics included overall accuracy, grade-specific accuracy, entity-specific accuracy, precision, recall, F₁-scores (per grade, weighted F₁ and macro F₁), and several measures of prediction error relative to expert grading. Accuracy, defined as the model’s agreement with the expert annotation, was calculated both across all cases and stratified by Clavien-Dindo grade, allowing assessment of whether model performance varied with complication severity. To account for potential heterogeneity across clinical indications, accuracy was also computed separately for each disease entity (HCC, CRLM, Klatskin tumors, and ICC). For this simplified task, we calculated overall and entity-specific accuracies, along with their corresponding confusion matrices, using ground-truth categories derived directly from the expert’s assigned Clavien-Dindo grades (Figure 4).

To quantify the magnitude of model-expert-disagreement, we calculated the absolute deviation, defined as the number of grade steps by which a model’s prediction differed from the ground truth. We report the distribution of deviation magnitudes, including the proportion of predictions differing by 1, 2, or more grade levels. These analyses were conducted overall and within each Clavien-Dindo grade, thereby enabling identification of grades that were particularly prone to over- or underestimation. In a final review step, outlier cases were defined as predictions deviating by more than 2 Clavien-Dindo grades from the expert-assigned reference and were manually reviewed. These cases were analyzed qualitatively to identify recurrent sources of disagreement, such as ambiguous documentation, implicit references to complication management, and inconsistent phrasing in physician letters. In addition, a majority voting approach was applied, aggregating predictions across the evaluated models and selecting the final grade based on the most frequent prediction. This ensemble strategy was used to assess whether combining model outputs could improve overall accuracy and reduce extreme disagreement with the expert annotation, under the assumption that individual models exhibit partially complementary error patterns, as suggested by the preceding error and deviation analyses.

For evaluating proprietary LLMs, a deidentified subset of 50 cases was created. This subset was intentionally balanced across the Clavien-Dindo grades, with 24 cases representing minor (0-IIIa) and 26 cases representing major (IIIb-V) complications. All documents underwent systematic deidentification [28], replacing patient names, dates, institutional references, and location information in alignment with institutional data protection guidelines. This subset was used exclusively for evaluating GPT-5 and Gemini 3 Pro, whereas open-weight models were evaluated on the full dataset of 650 cases.

Continuous variables were summarized as medians with IQRs. Before correlation analysis, normality of continuous variables was assessed using the Shapiro-Wilk test. As variables did not meet the assumption of normality, continuous clinical variables (eg, length of stay [LOS]) were analyzed descriptively and explored for association with complication severity using nonparametric rank-based correlation (Spearman ρ), given the ordinal structure of the Clavien-Dindo scale. To investigate potential sources of disagreement between model and expert annotation, logistic regression analyses with classification correctness (correct vs incorrect prediction) as the binary outcome variable were conducted. Document length, measured both in characters (per 1000 characters) and words (per 100 words), was included as the independent variable. Separate models were fitted for each evaluated LLM. Regression coefficients were exponentiated to yield odds ratios (ORs) with corresponding 95% CIs, quantifying the change in the odds of correct classification associated with incremental increases in document length. Statistical significance of individual predictors in logistic regression was evaluated using the Wald z test, and the significance level was set to P<.05 for all analyses. Interrater agreement between the 2 independent clinical annotators, as well as agreement between each model and the primary expert annotation, was quantified using Cohen κ, calculated overall and separately for each Clavien-Dindo grade. κ values were interpreted according to the commonly used benchmarks (<0.20 slight, 0.21-0.40 fair, 0.41-0.60 moderate, 0.61-0.80 substantial, and >0.80 almost perfect agreement). All analyses were conducted in Python 3.12.9 (Python Software Foundation) using the packages pandas (2.3.1), numpy (2.3.1), and scipy (1.16.3), following a prespecified analytical plan to ensure reproducibility.

The final dataset comprised 650 postoperative cases from 649 patients (n=229, 35% female) with a median age of 67 (IQR 58-73) years, treated between 2010 and 2024, and included 819 discharge letters. The cases cover the 4 major hepatobiliary disease entities: HCC (n=156, 24%), CRLM (n=113, 17%), perihilar cholangiocarcinoma (n=211, 33%), ICC (n=162, 25%), and other (n=8, 1%). The resulting expert annotation demonstrated a broad spectrum of postoperative complication severities within the study cohort. Most patients were classified as having no or minor complications, with grade 0 accounting for 334 (51%) cases, followed by grade I (n=90, 14%), grade II (n=63, 10%), and grade IIIa (n=63, 10%). Major complications were less frequent, including grade IIIb (n=41, 6%), grade IVa (n=11, 2%), grade IVb (n=6, 1%), and grade V (n=42, 6%). Overall, minor complications (grades 0-IIIa) comprised 85% (n=550) of the cohort, whereas major complications (grades IIIb-V) accounted for 15% (n=100; Table 2). The cohort characteristics of the 50-case subset are provided in Table S2 in Multimedia Appendix 1.

The physician discharge letters exhibited high variability in length, with a median of 4558 (IQR 3797-5849.75) characters and 547 (IQR 450-697.75) words. Document length showed a weak-to-moderate positive association with complication severity (Spearman ρ=0.39 for characters and ρ=0.37 for words; both P<.001). The median postoperative LOS was 13 (IQR 9-20) days and increased with higher Clavien-Dindo grades, demonstrating a moderate positive correlation with complication severity (Spearman ρ=0.48, P<.001). LOS was also moderately correlated with document length (Spearman ρ=0.35 for characters and ρ=0.32 for words; both P<.001), indicating that longer, more complex postoperative courses were accompanied by more extensive physician documentation. Given the distinct nature of Clavien-Dindo grade V, additional analyses excluding these cases are provided in Supplement S5 in Multimedia Appendix 1.

To contextualize model performance relative to human expert variability, interrater agreement between the 2 independent clinician annotations was assessed on the 65-case stratified subset. Overall, Cohen κ indicated substantial agreement between the 2 annotators (κ=0.752) for fine-grading and almost perfect agreement (κ=0.843) for binary grading, providing a human benchmark against which model-to-expert agreement can be interpreted. Grade-level results are presented in Table 3.

Agreement varied considerably across grades. Perfect agreement was observed for grade 0 (κ=1.000) and almost perfect agreement for grade V (κ=0.932) and grade I (κ=0.871), reflecting the relative clarity of these categories. Substantial agreement was found for grades II, IIIa, and IIIb (κ=0.681-0.719), while grades IVa and IVb showed the lowest interrater concordance (κ=0.405 and κ=0.476, respectively), consistent with the known ambiguity in clinical documentation. These findings are in line with previously reported interrater variability for Clavien-Dindo grading, where comparable levels of agreement for complex complication grades have been observed in urological surgery [29].

To evaluate the performance of proprietary LLMs under similar conditions, GPT-5.1 and Gemini 3 Pro Preview were prompted on a balanced, deidentified subset of 50 cases selected from the full dataset, ensuring uniform representation across all complication severities. While the 4 open-weight models were primarily evaluated on the full 650-case dataset as reported above, their performance was additionally assessed on the same 50-case subset to provide a methodologically equivalent basis for cross-model comparison. The full-dataset results for open-weight models and subset results for proprietary and open-weight models are not directly comparable. The subset-based results for both proprietary and open-weight models are presented side by side in Tables S11-S15 in Multimedia Appendix 1, including grade-level accuracy, F₁-scores, entity-specific accuracy, deviation distributions, and confusion matrices.

GPT-5.1 and Gemini 3 Pro each achieved an overall accuracy of 0.78, with macroaveraged F₁-scores of 0.766 and 0.751, and weighted F₁-scores of 0.774 and 0.761, respectively (Table 6). By comparison, open-weight models achieved overall accuracies of 0.60-0.70 on the same subset, with macroaveraged F₁-scores ranging from 0.557 to 0.673 and weighted F₁-scores from 0.570 to 0.688 (Table S11 in Multimedia Appendix 1).

Moreover, both proprietary models achieved perfect accuracy of 1.0 for grades I and V. The performance for grades IIIa and IIIb was also strong, with values ranging between 0.714 and 0.857 and F₁-scores from 0.714 to 0.833. In contrast, grade II showed mixed performance, with an accuracy of 0.6 and F₁-scores of 0.600-0.667, similar to the results of the open-weight models. For the rarer grades IVa and IVb, greater variability was observed, with accuracies ranging between 0.333 and 0.667 and F₁-scores between 0.444 and 0.667.

Overall, GPT-5.1 and Gemini 3 Pro demonstrated performance that was partly superior, partly comparable, and in some cases lower than the performance of the strongest open-weight models, depending on the complication grade and clinical entity. Both proprietary models showed clear strengths in several grades, including robust performance in grades I, IIIa, IIIb, and V, while exhibiting similar or slightly reduced performance in other grades, particularly some intermediate or low-frequency grades. When complications were grouped into minor (≤IIIa) and major (≥IIIb) categories, both proprietary models outperformed all open-weight models, achieving higher accuracy in distinguishing clinically relevant severity classes (Table S12 in Multimedia Appendix 1). Entity-level performance was consistently high for both proprietary models. Compared with open-weight baselines, proprietary models performed better or at least as well across all entities; particularly strong gains were observed in Klatskin cases, which involve complex postoperative trajectories and narrative heterogeneity (Table S13 in Multimedia Appendix 1). These results indicate that proprietary models generalize more robustly across diverse disease contexts.

In the binary classification task, proprietary models again achieved the highest performance. On the 50-case subset, GPT-5.1 reached an accuracy of 0.98 and a macroaveraged F₁-score of 0.980, with perfect recall for category A (1.00) and perfect precision for category B (1.00). Gemini 3 Pro Preview achieved an accuracy of 0.94 and a macroaveraged F₁-score of 0.940, with a recall of 0.958 for category A and 0.923 for category B. By comparison, open-weight models achieved accuracies of 0.90-0.94 on the same subset, with macroaveraged F₁-scores of 0.900-0.940 (Table S14 in Multimedia Appendix 1). Entity-level binary classification accuracy for GPT-5.1 and Gemini was uniformly high, ranging from 0.8 to 1.0 across all clinical subgroups, confirming that distinguishing minor from major complications is a highly tractable task for state-of-the-art LLMs (Table S15 in Multimedia Appendix 1).

To assess whether aggregating predictions across models could improve classification reliability, a simple majority-vote ensemble was constructed. For each case, the most frequently predicted Clavien-Dindo grade among all open-weight models was selected; ties were resolved by assigning the first occurring grade. Only valid numerical predictions were included in the voting process.

The majority-vote ensemble exceeds the performance of all individual open-weight and proprietary models in the fine-grained setting (Table 10). In the binary grading task, the ensemble reached an accuracy of 0.948, outperforming all individual open-weight models and matching the upper range of the proprietary systems. When expanded to include GPT-5.1 and Gemini 3 Pro Preview, the ensemble across all models achieved 0.794, the highest grade-prediction accuracy observed in the study for fine-grained grading, and maintained a high accuracy for binary grading.

Additionally, ensemble aggregation reduced large disagreement with expert annotation. Whereas 41 cases exhibited deviations of 3 or more Clavien-Dindo grades in individual model predictions, this number decreased to 14 cases when applying majority voting. This reduction indicates that ensemble-based decision-making effectively mitigates extreme errors by leveraging complementary strengths and partially offsetting individual model biases.

In this study, contemporary LLMs demonstrated promising accuracy in classifying postoperative complication severity according to the Clavien-Dindo system using routine discharge letters. Performance was consistently strong across open-weight and proprietary models, with substantial improvement for binary classification of minor versus major complications. No single model uniformly outperformed across all dimensions, highlighting the heterogeneous strengths of different architectures and the value of multimodel strategies. While proprietary models achieved the highest overall accuracy on the balanced 50-case subset, open-weight models demonstrated competitive performance with comparatively lower computational requirements, greater deployment flexibility, and enhanced data privacy compliance, which represent critical advantages for clinical implementation. Given the different dataset sizes used for model evaluation, cross-model comparisons are confined to the balanced 50-case subset. Notably, smaller models such as Ministral-3-8B performed comparably to much larger architectures, suggesting that resource-efficient alternatives may suffice for this task. Ministral-3-8B also exhibited a more conservative response strategy, producing the highest number of “not classifiable” outputs when encountering ambiguous information. While the impact on fine-grained classification was negligible, the effect was more pronounced in binary classification, where the accuracy was reduced by 5% when unclassified cases were treated as prediction errors. To contextualize model performance, interrater reliability between 2 independent clinical annotators yielded a Cohen κ=0.752. Model-to-expert agreement ranged from κ=0.541 to κ=0.663 across open-weight models, falling below but approaching this human benchmark, suggesting that LLM-based classification is operating close to the level of inherent clinician-to-clinician variability in Clavien-Dindo grading. Ensemble aggregation improved performance by leveraging complementary error profiles without requiring additional training or fine-tuning.

The observed association between Clavien-Dindo grades and length of hospital stay replicates the gradient originally reported by Dindo et al [6]. This concordance provides evidence for construct validity of the LLM-based scoring system in the present cohort and indicates that automated classification aligns with established clinical patterns. These findings extend previous work on automated complication grading. Staubli et al [30] reported high accuracy (30/31, 96%) using LLMs for Clavien-Dindo classification, but their study relied on clinical scenarios derived from literature rather than original patient documentation. Can et al [31] demonstrated that LLMs can extract structured variables from routine interventional oncology reports in hepatobiliary patients, with proprietary models outperforming open-weight alternatives in accuracy and longitudinal consistency. Our study confirms these patterns using real-world discharge letters, which contain unstructured narratives, variable documentation styles, and clinically relevant ambiguities absent from standardized vignettes. Importantly, when evaluated on the same 50-case subset, our results additionally demonstrate that open-weight models can reach competitive performance for specific tasks such as binary complication classification, suggesting that the performance gap between model classes may be task-dependent.

The Clavien-Dindo classification has become the most widely adopted framework for reporting surgical complications over the past 2 decades. Following amendments in 2009 [32], the authors reported 90% interrater agreement. However, empirical research on interrater agreement has remained limited, particularly regarding the effects of clinical experience, time constraints, or documentation quality. Studies in urological and head-and-neck surgery have reported only moderate inter-observer agreement for Clavien-Dindo scoring [29,33,34]. Dindo et al [6] further demonstrated significant differences in perceived complication severity between patients, nurses, and physicians. The LLMs evaluated in this study achieved higher consistency than reported human interrater reliability in subspecialty settings, suggesting that automated approaches may offer a more reproducible alternative for complication assessment in complex surgical cohorts.

Logistic regression analysis revealed that increasing document length was associated with lower classification accuracy, suggesting that longer discharge letters introduce narrative complexity rather than discriminative clarity. This underscores the value of concise, structured complication documentation for both automated classification and clinical information transfer. Similarly, determining whether postoperative interventions represent surgical complications or planned disease management proved challenging. Updated consensus guidelines now recommend classifying all postoperative interventions according to Clavien-Dindo regardless of presumed causality, acknowledging that definitive attribution is often clinically impossible [35]. The authors proposed organ-specific grading systems to address such liver-specific ambiguities, although this increases complexity and reduces cross-study comparability.

These findings have direct implications for clinical workflow integration and surgical outcomes research. LLM-based systems could automatically propose Clavien-Dindo grades after discharge documentation is completed, functioning as decision-support tools that reduce documentation burden while preserving physician oversight. In the envisioned deployment scenario, the treating clinician would receive an automated grade recommendation directly upon completion of the discharge letter and could either accept or modify the suggested grade before it is permanently recorded in the hospital information system. Thereby preserving clinical authority while reducing manual abstraction effort and contributing to a continuously growing, structured outcomes database. This approach embeds automated grading as a decision-support artifact that augments rather than replaces clinical judgment, consistent with evidence that machine learning–based clinical decision support systems achieve high clinician acceptance when designed to preserve physician oversight [36,37]. To mitigate the risk of high-impact misclassifications in clinical deployment, we propose that document characteristics associated with elevated misclassification risk, such as increased discharge letter length or the presence of ambiguous procedural terminology, should serve as additional flags for prioritized human oversight. For broader adoption in clinical practice, AI-derived classifications will ultimately require structured and interoperable data representations. HL7 FHIR provides a potential framework for integrating such outputs into clinical information systems [23]. As implementation aspects were beyond the scope of this study, further considerations are provided in Figure S3 in Multimedia Appendix 1.

Beyond real-time use, automated grading is particularly well-suited for retrospective research. Large surgical cohorts can be efficiently annotated when complication grading was not performed systematically at the time of care, addressing a long-standing bottleneck in outcomes research. By bridging unstructured clinical narratives and structured metrics, LLMs offer scalable, reproducible quality assessment that addresses interrater variability and practical constraints limiting broader adoption of systematic complication scoring.

Despite overall strong performance, a subset of cases demonstrated significant deviations between LLM predictions and expert assessments, reflecting a fundamental tension in the Clavien-Dindo system: classification depends on the therapy required, yet the clinical significance of laboratory abnormalities without intervention remains subject to interpretation, a judgment often not explicitly documented in discharge letters. More broadly, it should be acknowledged that all classifications, both by human annotators and LLMs, are based exclusively on the information documented in discharge letters, which may not fully reflect the true clinical course. Documentation quality, narrative style, and individual reporting practices may influence the assigned grade, and the system therefore assesses documented rather than verified clinical outcomes. While this does not introduce differential bias in the study design of this study, it represents an inherent limitation of documentation-based outcome classification that should be considered when interpreting results in the context of real-world quality monitoring. Future work should explore richer input representations, including operative notes, progress notes, laboratory values, and imaging reports, to assess the added value of longitudinal documentation for complication grading.

Furthermore, expert annotation was performed by a single senior surgeon, which may introduce individual bias in the assignment of Clavien-Dindo grades. To partially address this limitation, a stratified 10% subset was independently annotated by a second clinician. Complete dual annotation of all 650 cases was not feasible due to the substantial time investment required for expert-level Clavien-Dindo annotation of complex hepatobiliary discharge letters. Disagreements between the 2 annotators were reviewed but did not inform retrospective refinement of the primary annotations, as selectively revising only a subset of the reference standard would have introduced a different form of bias. Formal consensus adjudication was considered but not implemented, as it would have required a third independent expert annotator and represents an avenue for future work. However, interrater agreement was assessed on this subset only and cannot be assumed to be representative. The remaining cases rely solely on single-annotator classification, and observed model-to-expert nonagreement in these cases may therefore partly reflect individual annotator variability rather than true model error. Future studies should incorporate multiple annotators with varying levels of clinical experience to establish a more robust reference standard. In addition, the cases originated from a single care center, reflecting center-specific documentation styles, perioperative pathways, and clinical workflows. Performance may therefore differ in multicenter settings or institutions with alternative reporting standards [38]. At the same time, the 14-year observation period introduces the possibility of temporal changes in documentation practice, which may influence the model performance over time. Multicenter validation studies are essential to assess generalizability and transferability across different clinical contexts.

A further methodological limitation concerns the comparability of open-weight and proprietary model evaluations. Open-weight models were evaluated on the full 650-case dataset, while proprietary models were assessed on a balanced 50-case subset due to cost and access constraints. Although all models were additionally evaluated on the same 50-case subset to enable direct comparison, the difference in evaluation scope limits the interpretability of cross-model conclusions drawn from the full dataset results. All models were evaluated in a zero-shot setting, without task-specific fine-tuning or domain adaptation, thereby limiting performance to intrinsic out-of-the-box reasoning capabilities. Accordingly, the results of this study should be interpreted as a reproducible baseline assessment rather than an optimized deployment scenario. While this choice enables a fair and transparent comparison across models, future work should explore fine-tuning and few-shot prompting, and retrieval-augmented approaches. These methods represent promising strategies for improving performance on challenging grades such as IVa and IVb, where zero-shot performance remained poor across all models. In particular, encoder-only architectures specifically optimized for text classification tasks may represent a more resource-efficient alternative to large generative models and still have high performance. Concurrently, aggregation methods that extend beyond simple majority voting (eg, LLM-as-a-judge frameworks or model councils) have the potential to enhance reliability and clinical alignment by explicitly resolving disagreements, particularly in cases of borderline or rare complication grades. In this context, the robust performance of Ministral-3-8B indicates that further investigation is warranted into SLMs. Encoder-based architectures optimized for text classification may represent a more resource-efficient alternative, with reduced computational cost, inference latency, and ease of integration into clinical IT infrastructures. Nevertheless, the generalizability of these findings is constrained by the size and scope of the dataset. Larger, more diverse, multicenter datasets spanning additional surgical domains are needed to enhance robustness, particularly for rare complication grades. Furthermore, prospective validation in real-world clinical workflows remains a critical next step. Such studies are necessary to evaluate practical utility, assess effects on documentation burden and clinical decision-making, and identify potential biases introduced by automated complication grading systems. Furthermore, although all models were prompted to provide a cited passage as an interpretability basis for their grade assignments, citation accuracy was not formally evaluated and should be considered indicative rather than verified, representing an important direction for future research.

This study shows that contemporary LLMs can reliably classify postoperative complications according to the Clavien-Dindo system using routine clinical documentation. Open-weight models offer a particularly attractive trade-off by combining competitive accuracy with substantially lower computational demands, while ensemble strategies further enhance robustness. Notably, the performance of the best open-weight models approached the upper bound of human interrater agreement, suggesting that remaining discrepancies between model and expert annotation may partly reflect the inherent variability of clinical grading rather than model limitations alone. Overall, these results highlight the potential of LLMs as a useful tool for supporting surgical complication assessment at scale.