The study design is summarized in Figure 1. This study was reported in accordance with the TRIPOD-LLM (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis-Large Language Model) [20]. A completed checklist is provided in Checklist 1.

We developed the benchmark using the HIRA reimbursement review guidelines for anticancer drugs, version dated February 1, 2026. Three major gynecologic cancers—cervical cancer, uterine cancer, and ovarian cancer—were included. For each cancer type, reimbursement criteria were translated into structured clinical attributes representing heterogeneous patient scenarios. Candidate benchmark items were created by a researcher with expertise in medical informatics (JS).

The benchmark was constructed within a tristate adjudication framework, including eligible, ineligible, and undeterminable outcomes for each anticancer regimen. This outcome assignment is based on 3 different condition-level states: met, not met, and unevaluable. We use “not met” to denote a condition whose required value is present but violates the criterion, and “unevaluable” to denote a condition that cannot be assessed because the required information is absent (Figure 2). The unevaluable state was included to reflect real-world documentation contexts in which information required for applying a coverage criterion may be absent, incomplete, or inaccessible.

For instance, a case is classified as ineligible when 1 or more conditions are explicitly not met, regardless of whether other conditions are unevaluable. A case is classified as undeterminable when no condition is explicitly not met, yet 1 or more conditions remain unevaluable. To control task difficulty, ineligible and undeterminable cases were designed as near-miss scenarios containing only 1 to 2 conditions not met or unevaluable. The detailed definitions of the outcomes are shown in Table 1.

The benchmark was reviewed by 3 gynecologic oncology experts, including 2 nurse practitioners and 1 physician. Validation focused on the clinical plausibility of the patient scenarios, the appropriateness of attribute combinations, and the consistency of the reference answers.

For instance, the experts identified that palliative regimens inherently target patients with recurrent or metastatic disease, meaning that clinical indication is implicitly satisfied, and its removal alone does not produce a valid undeterminable case. For such regimens, additional clinical attributes had to be removed to construct undeterminable scenarios. The review also identified certain regimens that are permissively reimbursed in clinical practice despite not strictly meeting the guideline criteria. For these cases, reference answers were assigned based on the guideline criteria.

After clinical expert review, all benchmark cases were independently reviewed by a utilization review nurse, and discrepancies were resolved through consensus while preserving the balanced class structure. Interrater agreement between experts and the utilization review nurse was assessed using Cohen κ. This multidisciplinary validation approach reflects real-world reimbursement review processes, which involve both clinical and administrative expertise.

We evaluated 6 LLMs from 3 providers: Gemini 3.1 Pro (gemini-3.1-pro-preview) and Gemini 3 Flash (gemini-3-flash-preview) from Google; Claude Opus 4.6 (claude-opus-4‐6) and Claude Sonnet 4.6 (claude-sonnet-4‐6) from Anthropic; and GPT-5.4 (gpt-5.4-2026-03-05) and GPT-5 Mini (gpt-5-mini-2025-08-07) from OpenAI. All models were evaluated under the primary condition, in which the original HIRA guideline PDF was provided via each model’s native document upload feature. Guideline documents were provided as 3 cancer-specific PDF files corresponding to cervical, uterine, and ovarian cancer. Because the set of user-adjustable hyperparameters differed across models and the evaluation included both reasoning and nonreasoning models, all models were evaluated using provider-default settings, and no parameter was explicitly modified. To assess response stability, each model was run 3 times under the same condition.

All models received the same standardized prompt template (Textbox 1). The system prompt instructed the model to act as an expert reviewer of Korean NHI reimbursement for oncology regimens and to return a structured JSON response containing a 3-class decision (eligible, ineligible, or undeterminable) and a single-sentence rationale. The user prompt specified the cancer type, regimen name, and structured clinical and administrative attributes.

Each model was evaluated over 3 independent runs. To capture run-to-run consistency of the models, accuracy was recorded to compute mean and SD. A final predicted label for each case was determined by majority voting across the 3 runs. A 3-way tie was resolved as undeterminable, reflecting the model’s own uncertainty about the case.

All evaluation metrics—precision, recall, and F₁-score—were calculated from these majority-voted predictions. Ninety-five percent CIs for accuracy, precision, and recall were calculated by Wilson method, and those for the F₁-score were obtained by bootstrapping (n=1000). The Bhapkar test of marginal homogeneity was used to assess whether the predicted outcome distribution differed from the observed distribution across the 3 outcome classes. Analyses were performed using Python 3.10 with SciPy.

We further conducted a retrospective error analysis of all misclassified cases to characterize how models failed across the benchmark. Because the benchmark was explicitly designed around a tristate adjudication framework, model errors could be interpreted according to the adjudication principle they violated.

We therefore conducted a logic-based error analysis in which each expected-predicted outcome pair was mapped to an error category. Representative rationales and guideline locations were then reviewed to characterize recurrent structural sources of the dominant error category. This rationale review was performed blinded to model identity to mitigate potential reviewer bias.

Sensitivity analysis consisted of 3 strategies. First, to examine the effect of input format, the guideline file (PDF) was parsed into structured text (Markdown) using the pdfplumber library, and all models were evaluated with the extracted text as input. Second, to assess the effect of information access, search-enabled conditions were evaluated on all models. This setting was intended to reflect scenarios in which patients or health care providers use LLMs with web search enabled. Third, to investigate the effect of prompting strategy, a prompt informed by the error analysis findings was evaluated, instructing models to verify each eligibility condition step by step.

For the sensitivity analysis, each prediction was dichotomized as correct or incorrect against the reference standard. Pairwise comparisons of correctness between the baseline and each sensitivity condition were then performed using McNemar test, with Benjamini-Hochberg correction for the false discovery rate. Adjusted P values are reported as q values.

This study did not involve human participants, real patient data, or protected health information. All patient scenarios were synthetically constructed based on publicly available reimbursement guidelines published by the HIRA in South Korea.

Agreement between the utilization review nurse and gynecologic oncology experts was initially 95% (211/222; Cohen κ=0.93). Eleven cases were identified in which specific qualifying conditions had not been fully reflected in the benchmark scenarios. These discrepancies were resolved through consensus while preserving the balanced class structure.

The final benchmark composition is summarized in Table 2. A total of 74 anticancer regimens (15 cervical, 17 uterine, and 42 ovarian) were each assigned 3 outcome scenarios, yielding 222 cases. The mean number of eligibility conditions per regimen was 4.2 (SD 1.9; range 2‐11). Uterine (mean 4.5, SD 2.8; range 3-11) and ovarian cancer regimens showed comparable mean complexity (mean 4.5, SD 1.3; range 2-8 for both), but uterine cancer included the most complex individual regimens, with up to 11 eligibility conditions. Cervical cancer had the simplest criteria (mean 2.9, SD 1.4; range 2‐6).

The upper range of complexity in uterine cancer was largely driven by immunotherapy regimens such as pembrolizumab plus lenvatinib, which required up to 11 simultaneous conditions, including deficient mismatch repair or microsatellite instability-high status, Eastern Cooperative Oncology Group performance score, and prior treatment history.

Overall accuracy ranged from 77.9% (GPT-5.4) to 88.7% (Gemini 3.1 Pro) across the 6 models. No malformed outputs were observed. Performance was relatively stable across runs, with SDs ranging from 0.3 to 2.0 percentage points (pp). Despite these differences in overall accuracy, all models showed the same general pattern of substantially lower recall for undeterminable cases than for eligible or ineligible cases. Three-way ties across runs were rare, occurring in only 1 case out of all model evaluations (222 cases × 6 models). Detailed results are provided in Multimedia Appendix 1.

The LLM output was analyzed by 3 outcome classes (Figure 3). For eligible cases, recall was high across models (93.2%‐98.6%), and ineligible cases showed similarly strong performance (86.5%‐97.3%). By contrast, undeterminable cases showed a marked decline across all models, ranging from 44.6% to 70.3%. Among the models, Gemini 3.1 Pro showed the highest recall for undeterminable cases (70.3%), whereas GPT-5.4 showed the lowest (44.6%). The full results with precision, recall, F₁-score, and confusion matrix are provided in Multimedia Appendix 2.

Analysis of misclassification direction for undeterminable cases (Table 3, 44.6%‐70.3%) revealed that models predominantly misclassified these cases as eligible (28.4%‐50%) rather than ineligible (0%‐5.4%). This pattern was observed across all 6 models. Consistent with this pattern, Bhapkar tests showed significant marginal heterogeneity between observed and predicted outcome distributions for all models (all P<.001; Multimedia Appendix 3).

Performance varied across cancer types. Cervical cancer showed the highest mean recall across 6 models (92.6%), followed by ovarian (82.9%) and uterine (71.9%) cancer. Per-regimen complexity was lowest for cervical cancer (mean 2.9, SD 1.4; range 2‐6 conditions) and highest for uterine cancer (mean 4.5, SD 2.8; range 3‐11). Notably, recall varied substantially by outcome class within each cancer type (Table 4). While eligible and ineligible recall remained high across cancer types (89.7%‐100%), undeterminable recall dropped sharply, particularly for uterine cancer (mean 16.7%), where per-regimen complexity was highest. Model-wise results by cancer type and outcome class are provided in Multimedia Appendix 4.

The tristate benchmark design enabled a logic-based characterization of model errors. All 235 misclassified cases were assigned to 1 of 3 error types according to how each misclassification violated the adjudication logic; predicted outcomes and single-sentence rationales were reviewed to characterize each category (Table 5).

The most common error was information gap-filling (196/235, 83.4%), in which models inferred a definitive outcome from incomplete scenarios rather than recognizing missing information. The second most common error type was criterion misapplication (n=20, 8.5%), in which models classified cases as eligible despite 1 or more clearly not-met conditions or, conversely, classified cases as ineligible despite all conditions being met. Third, false uncertainty (n=19, 8.1%) refers to cases where models inferred undeterminable despite sufficient information to adjudicate eligible or ineligible. This pattern occurred most frequently in ineligible scenarios where 1 or more conditions were unevaluable but at least 1 clearly unmet condition was present.

To further understand the dominant error type, information gap cases were analyzed in relation to guideline structure. Reimbursement guidelines are organized hierarchically (Textbox 2), and model performance varied with structural complexity. In particular, subheading (L2) and footnote (L4) were the weak points of the models.

The largest subtype involved models ignoring the document subheading (L2) that specifies therapeutic intent (eg, palliative, maintenance). This distinction is clinically significant because identical regimens may appear under multiple therapeutic purposes with different indications, and reimbursement is granted only for the purpose under which the regimen is listed.

Another subtype involved footnote (L4) recognition. Designated institution requirement for immune checkpoint inhibitor, restricting eligible prescribing sites to those with qualified specialist staffing, appeared only in guideline footnotes, and most models systematically missed this condition.

Building on the failure modes identified in the error analysis, 3 sensitivity conditions were evaluated (Multimedia Appendix 5). These analyses included structured text, web search–enabled, and structure-guided prompt conditions.

In the structured text condition, the parsed guideline in Markdown format was provided to the models. Performance declined across all 6 models, with statistically significant degradation observed in the 2 Claude models (Claude Opus 4.6: Δ=−10.4 pp; Claude Sonnet 4.6: Δ=−10.8 pp; both q<.001). The remaining 4 models showed smaller declines that did not reach significance.

The web search–enabled condition assessed the effect of additional information access alongside the guideline document, while the structure-guided prompt condition incorporated the hierarchical levels of the guideline (Textbox 2) into the prompt and instructed models to verify each eligibility condition step by step against the corresponding level. Neither condition produced a statistically significant change from baseline in any model (all q>.05).

Notably, web search tool invocation rates ranged from 0% to 3.2% across the 6 models under the web search–enabled condition. Token usage nevertheless increased substantially: OpenAI models showed approximately 4500 additional input tokens per case from the embedded tool specification, while Claude models produced longer reasoning.

This study developed a benchmark for anticancer drug reimbursement eligibility determination covering 3 gynecologic cancers and used it to evaluate 6 LLMs from 3 providers. The benchmark was based on a tristate adjudication framework that distinguishes condition-level states (met, not met, unevaluable) and case-level outcomes (eligible, ineligible, and undeterminable) for eligibility verification. Case-level outcomes are derived from the aggregation of condition-level states.

The principal findings are threefold. First, overall accuracy varied across models, ranging from 77.9% to 88.7%, indicating broadly comparable performance across the 6 evaluated models. Second, performance varied sharply by outcome class. While eligible and ineligible cases were classified with consistently high recall, undeterminable cases posed the greatest challenge across all models, revealing a consistent limitation to recognize when clinical information is insufficient for a determination. Third, the tristate benchmark design enabled a logic-based characterization of model errors. Rather than treating all incorrect predictions as equivalent, the framework distinguished information gap-filling, criterion misapplication, and false uncertainty according to how each misclassification violated the expected adjudication logic.

These findings suggest that LLMs may help reduce the manual burden of eligibility review for clearly eligible and ineligible cases. However, these models should be positioned as decision-support tools operating under human oversight, not as autonomous decision-makers. The systematic failure in undeterminable cases represents the most clinically consequential limitation, as models tend to resolve ambiguity toward eligibility rather than deferring judgment when information is incomplete.

The dominant undeterminable failure pattern—information gap-filling—has direct clinical implications. When models misclassified undeterminable cases, the predominant direction was toward eligible rather than ineligible, indicating a tendency to infer eligibility from incomplete information. The severity of this pattern varied by cancer type, corresponding to the number of eligibility conditions per regimen. Uterine cancer regimens, which had the largest mean number of eligibility conditions, showed the lowest undeterminable recall.

This tendency has potential financial implications for hospitals. Under South Korea’s retrospective claims review structure, ineligibility identified during postadministration audit can result in full denial of previously reimbursed drug costs, with the burden shifted to the prescribing institution. Because high-cost agents such as immune checkpoint inhibitors, antiangiogenic agents, and poly (ADP-ribose) polymerase inhibitors are reimbursed under exception-coverage provisions for severe diseases, even a small number of misclassifications can accumulate into substantial losses. Given that per-cycle costs typically range from KRW 3‐6 million (approximately US $2200‐$4400), cumulative nonrecoverable costs can reach tens of millions of KRW per patient. Although the per-case inference cost itself is minimal (approximately US $0.002‐$0.10 across the 6 models), an erroneous determination at this cost can still translate into substantial financial losses through retrospective denial.

Meanwhile, sensitivity analysis revealed 3 patterns. First, converting the guideline file to structured text degraded performance across all models, indicating that document structure conveys eligibility logic lost during text extraction. Second, enabling web search did not produce a statistically significant change from baseline for any model (all q>.05), with models defaulting to the provided guideline rather than calling the search tool. This pattern indicates that in realistic scenarios where authoritative source documents are accessible to the LLM, the models rely on the provided context, and enabling the search tool neither improved nor degraded outcomes while increasing inference cost. Third, structure-guided prompting did not produce a statistically significant improvement for any of the 6 models (all q>.05), indicating that explicit structural guidance combined with step-by-step per-condition verification was insufficient to overcome the inherent limitations in hierarchical guideline reasoning observed in this benchmark.

Taken together, these results indicate that current LLMs are not yet reliable as standalone tools for reimbursement eligibility determination, particularly when clinical information is incomplete. Because neither input reformatting, information augmentation, nor prompt modification alone reliably improved performance, more systematic integration strategies will be needed to deploy LLMs safely in reimbursement workflows.

Most LLM benchmarks in the medical domain have focused on knowledge assessment through multiple-choice examinations such as MedQA [21], while real-world task evaluation remains scarce [22]. Recent frameworks such as MedHELM [23] have begun addressing this gap but reveal that administrative and workflow tasks remain the weakest evaluation category.

Within guideline-based reasoning, MedGUIDE [24] and CPGPrompt [25] evaluated LLM adherence to cancer treatment decision trees and referral classification guidelines, respectively, both finding that models frequently deviate from structured conditional logic. Outside the medical domain, RuleArena [26] reported similar difficulties with complex rule-guided reasoning. However, none of these benchmarks test whether models can recognize when information is insufficient for a determination; all evaluate scenarios in which sufficient information is provided and a definitive answer exists.

This limitation is directly relevant to our findings. Recent abstention studies—AbstentionBench [27] across 20 datasets and MedAbstain [28] in medical question answering—showed that providing an explicit abstention option consistently increased model uncertainty and safer abstention behavior, whereas scaling model size or applying structure-guided prompting yielded little improvement.

Our undeterminable outcome class parallels these abstention scenarios in a domain-specific context. However, whereas these benchmarks test whether models can detect missing information within a self-contained question, our benchmark additionally requires models to extract eligibility conditions from a hierarchically structured guideline document and then determine whether the patient scenario provides sufficient information to evaluate each condition.

To our knowledge, no prior benchmark has evaluated LLM performance on health insurance reimbursement eligibility. This study addresses this gap by introducing a benchmark grounded in national anticancer drug coverage criteria with expert validation, incorporating an undeterminable outcome class that explicitly tests recognition of incomplete information.

This study has several limitations. First, the benchmark was limited to 3 gynecologic cancers based on South Korean NHI guidelines and relied on a relatively small dataset. Its generalizability to other cancer types, guideline versions, or insurance systems requires further validation.

Second, all patient scenarios were synthetically constructed in a structured format. This design does not capture the additional uncertainty introduced by extracting relevant attributes from unstructured records. Because real-world clinical documentation is often incomplete and unstructured, the performance observed in this study may overestimate model utility in practice.

Third, some eligibility conditions appeared in the global caveat and footnotes; errors involving these structural elements may partly reflect document representation rather than reasoning limitations of models alone.

Fourth, our benchmark adopts a strict rule that treats any missing information required by guideline criteria as undeterminable. This reflects a fundamental distinction between clinical reasoning and administrative reasoning. For instance, the absence of a prior treatment record may suggest a patient who has not previously been treated in clinical reasoning but constitutes missing evidence in administrative adjudication. Findings should, therefore, be interpreted as specific to administrative reasoning tasks.

Fifth, while the qualitative error analysis was conducted with model identities blinded, LLM outputs often contain stylistic fingerprints (eg, characteristic phrasing or cadences) that may make perfect blinding difficult. Residual unblinding may, therefore, introduce some bias into our error categorization.

Finally, models were tested with provider-default settings without fine-tuning or agentic frameworks, which may underestimate achievable performance.

Several directions arise from these limitations. Future work should expand the benchmark to additional cancer types and guideline versions and validate it using real clinical documentation to better reflect performance under unstructured and incomplete records. A direct human performance baseline, in which experts solve the same cases under identical conditions, would also be valuable to contextualize LLM accuracy against expert-level performance.

Methodologically, agentic approaches that iteratively verify each condition against the guideline may improve accuracy on complex hierarchical structures. In addition, hybrid neuro-symbolic architectures that pair LLM-based understanding with structured rule engines have shown initial promise in adjacent health care tasks [29]; their applicability to reimbursement reasoning under incomplete information, particularly in handling rigid AND/OR logical constraints, remains to be evaluated.

The tristate adjudication framework introduced in this study offers a distinctive approach to evaluating LLM behavior in logical reasoning contexts characterized by variable information completeness. In this benchmark, LLMs classified clearly eligible and ineligible cases with relatively high recall but showed limited reliability on undeterminable cases. The dominant error pattern was information gap-filling, in which models tended to infer eligibility rather than withhold judgment. These findings indicate that LLMs, in their current form, should be deployed as supervised decision-support tools rather than as independent adjudicators in reimbursement review.