This study was approved by the Institutional Review Board of Seoul National University Hospital (approval number 2401-084-1501) and was conducted in accordance with the Declaration of Helsinki. Written informed consent was obtained from all participants.

From April 2024 to May 2025, adults (≥18 y) receiving oncologic care were recruited consecutively from outpatient and inpatient oncology services at 3 secondary and tertiary hospitals in South Korea. The exclusion criteria were medical, cognitive, or linguistic limitations that precluded meaningful interviews.

Interviews were conducted by 3 psycho-oncology psychiatrists using a custom application that enabled real-time symptom scoring and time-stamped tagging. During the interviews, psychiatrists assigned ratings in real time using the application, with each entry automatically time-stamped. This design created a direct link between the clinical judgment and the corresponding segment of the audio recording, producing an annotated dataset that aligned symptom scores with a conversational context.

The interviews were conducted in a semistructured format, comprising 39 evaluation items across four domains: (1) psychiatric symptoms (29 items), (2) physical symptoms (4 items), (3) psychiatric interference (3 items), and (4) physical interference (3 items). Each item was rated on a 4-point Likert scale (0‐3), with higher scores indicating greater severity or interference.

All interview recordings were transcribed using speech-to-text (STT) software, manually corrected, and time-aligned by trained assistants without a medical background. All data were fully deidentified before the analysis.

To assess clinician reference rating reliability, an independent board-certified psychiatrist, blinded to the original ratings, reevaluated 2 complementary samples: 15 high-distress cases (≥12 clinically significant items) and an additional 15 cases sampled without restriction on distress severity. Interrater agreement was high in both the high-distress sample (intraclass correlation coefficient [ICC]=0.904) and the broader-severity sample (ICC=0.899). Agreement was also high in the pooled 30-case analysis (ICC=0.911), supporting the use of these ratings as a reference standard [20].

An LLM-based evaluation was conducted from June to August 2025 using APIs. Three LLMs were evaluated: GPT-4o (OpenAI), Claude 3.5 Sonnet (Anthropic), and Gemini 2.5 Flash (Google DeepMind), with a temperature of 0.3, a maximum token limit of 20,000, and default parameters otherwise.

All interview recordings, transcripts, clinician ratings, and adjudication data remained nonpublic throughout the study period and were not publicly released, posted online, deposited in public repositories, included in preprints, or shared as publicly accessible materials before or during the LLM evaluation. Only deidentified transcripts and item-specific evaluation inputs were used for API-based LLM inference. Therefore, contamination through publicly available training materials was considered highly unlikely.

For each patient-model combination, the complete Korean transcript and corresponding time-stamped item-specific windows were submitted in a single API call. The models were instructed to assign ordinal scores (0‐3) to each of the 39 items and generate brief textual justifications for each rating. If a call timed out or did not return a complete response conforming to the prespecified JSON schema, the same request was repeated, and the first complete schema-conforming response was retained. No additional generation was performed for that patient-model combination.

All models were guided by an identical zero-shot prompt in Korean that combined explicit evaluation rubrics with schema-constrained outputs and reasoning justifications. The prompt assigned models the role of a psychiatrist specializing in psycho-oncology. The instructions specified a fixed JSON structure with predefined keys for all items and required ratings on a standardized 4-point rubric anchored to symptom frequency, duration, and functional impairment over the preceding 1 to 2 weeks. The distinction between scores of 0 to 1 (subclinical) and 2 to 3 (clinically significant) was emphasized as the threshold for clinical significance. Conservative scoring was operationalized within the prompt by requiring clear transcript evidence for clinically significant ratings and by instructing models to assign lower scores when evidence regarding symptom frequency, duration, or functional impact was unclear or insufficient. The prompt also included interpretation guidelines to account for potential STT errors, such as phonetic misrecognitions, fragmented speech, and indirect emotional expressions.

During prompt development, we evaluated 2 rubric formats. One format used a single standardized 4-point rubric applied uniformly across all symptom items, whereas the other provided symptom-specific descriptions for scores 0, 1, 2, and 3 for each item. The standardized rubric format was selected for the final evaluation because pilot review showed more consistent application of shared scoring criteria across items, including symptom frequency, duration, and functional impact. We then compared 2 input-context formats. In the first, the model received only the transcript excerpt corresponding to each tagged symptom. In the second, the model received the complete interview transcript together with the item-specific excerpt (ie, the portion of the interview where the symptom was mentioned). The latter format was selected to preserve the broader clinical context of the interview.

These development steps concerned the general rubric and contextual input structure rather than model-specific prompt tuning. The final prompt was fixed before the main evaluation and applied identically across all models. No model-specific prompt modifications or few-shot demonstrations containing example transcripts, completed item ratings, or model-output examples were used. The only examples included in the prompt were generic anchor expressions embedded in the scoring rubric. An English translation of the full prompt is provided in Multimedia Appendix 1.

Binary mismatches (clinically significant symptoms defined as score ≥2) were manually reviewed and classified using a taxonomy developed and refined through consensus within a multidisciplinary team that included clinical experts and data scientists. Selected cases were reviewed in multidisciplinary team meetings for calibration of the operational definitions and consistency of taxonomy application. A single psychiatrist served as the primary adjudicator and performed the case-level classification with explicit model-source labels masked. For each mismatch, the adjudicator reviewed the target item, the deidentified transcript and item-specific segment, the clinician and model ratings, and the model-generated rationale.

Disagreements were categorized as ambiguous (insufficient or equivocal transcript evidence) or definite overestimation or underestimation, with directional errors further classified into 8 etiologies (Table 1). Reproducibility was assessed through an independent, model-blinded review of 150 mismatches stratified by model and initial adjudication category, yielding high interrater agreement for the ambiguous-versus-definite classification (Cohen κ=0.82). To provide a clearer understanding of ambiguous disagreements, 3 deidentified and translated examples are presented in Multimedia Appendix 2.

To systematically investigate the qualitative characteristics of the LLM outputs, an automated meta-evaluation framework was applied to all model-generated responses between June and July 2026. To reduce dependence on any single evaluator model family, model-generated rationales were independently meta-evaluated by GPT-5.4, Claude Sonnet 4.6, and Gemini Pro 3.1 using the same rubric, with explicit source-model identifiers removed. Reasoning outputs were assessed across 4 dimensions: citation (specificity of referenced evidence), structure (evidence, interpretation, and score completeness), mapping (rules and consistency between interpretation and score), and expansion (extrapolation beyond the transcript). For each rationale and dimension, the arithmetic mean of the 3 independently assigned scores was used as the meta-evaluation score. In a 10-patient subset, agreement between the mean meta-evaluation scores and psychiatrist ratings ranged from ICC (A,1)=0.646 to 0.822 across dimensions. Detailed reliability methods and results are provided in Multimedia Appendix 1.

The meta-evaluation prompt was developed independently of the primary symptom-rating prompt. During prompt development, the rubric was refined through pilot review to improve interpretability and reduce common scoring artifacts, including over-rewarding verbose rationales, penalizing appropriate documentation of symptom absence, or relying on keyword-based scoring. After pilot review, the prompt was finalized and applied uniformly to the full set of model-generated rationales. The parameters were a temperature of 0.15 and a maximum token count of 131,702. An English translation of the full prompt is provided in Multimedia Appendix 1.

Statistical analyses were performed using Python (version 3.12; libraries: NumPy, SciPy, pandas, scikit-learn, and statsmodels). All hypothesis tests were 2-sided, with the threshold for statistical significance set at P<.05. To address the multiplicity of related tests (patient-level directional bias, meta-feature comparisons, and mixed-effects models), P values were adjusted using the Benjamini-Hochberg false discovery rate. For 2-df Wald tests comparing source-model-specific ICCs across the 4 meta-evaluation dimensions and the post hoc ambiguity analyses, the Holm procedure was applied.

Because this study was designed to estimate model agreement, directional bias, and error patterns rather than to test a single confirmatory hypothesis, all consecutively recruited eligible participants during the study period were analyzed. Although each interview contributed up to 39 item-level evaluations, these observations were clustered within patients and were not treated as independent patient-level cases. To assess the stability of key performance estimates, we calculated 95% CIs using nonparametric patient-level cluster bootstrapping with 2000 resamples. In each bootstrap resample, patients rather than individual item-level observations were sampled with replacement, and all item-level observations from each selected patient were retained.

Descriptive statistics were used to summarize the participant demographics.

The concordance between LLM outputs and psychiatrist ratings on a 0 to 3 ordinal scale was assessed using quadratic weighted kappa (QWK) and ICC (2,1). The exact accuracy was also reported. For screening performance, scores were dichotomized (0‐1 vs ≥2) to compute sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and F₁-score. As a sensitivity analysis, the 170 patient-item pairs classified as ambiguous for any of the 3 models were excluded identically from all models, and the binary screening metrics were recalculated. The patient-level directional bias was evaluated by comparing the total clinically significant symptom counts per patient using the paired Wilcoxon signed-rank test.

Post hoc exploratory analyses of ambiguous mismatches were performed to examine whether the proportion of adjudicated mismatches classified as ambiguous differed across models or clinical domains. A total of 904 binary mismatches were analyzed using patient-clustered logistic generalized estimating equation models, which accounted for multiple mismatch observations from the same patient. Ambiguity status (ambiguous vs definite error) was the outcome, and model and clinical domain (psychiatric symptoms, physical symptoms, psychiatric interference, and physical interference) were included simultaneously as predictors. Following the global model test, pairwise model contrasts were estimated. To examine whether model differences varied across clinical domains, we tested the model-by-domain interaction. To examine whether model differences varied according to patient characteristics, separate domain-adjusted models tested interactions between model and standardized age, standardized clinician-rated symptom burden, major depressive episode status, and sex.

To examine whether reasoning styles differed across the models, the distributions of the meta-evaluation scores were compared using the Friedman test. To evaluate associations between rationale characteristics and prediction error while accounting for cross-classified data structures, mixed-effects models were fitted with the absolute error as a dependent variable. Meta-evaluation scores were included as fixed effects and were z-standardized within each model. Clinical domain (psychiatric symptoms, physical symptoms, psychiatric interference, and physical interference) was included as a 4-level fixed effect to account for systematic differences in error across domains. Random intercepts were specified for patients and items to account for within-patient clustering and item-level heterogeneity, respectively. The models were estimated using restricted maximum likelihood. As a sensitivity analysis, the same model specification was applied to the 10-patient subset using the human reference meta-evaluation scores. The resulting coefficients were compared with those obtained using the mean scores assigned by the 3 evaluator models in the same 10-patient subset and in the full cohort.

The TRIPOD-LLM (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis–Large Language Model) reporting guideline was used to guide the reporting of this study. The completed checklist is provided in Checklist 1 [21].

This study included 101 patients receiving oncology treatment or follow-up care. Each interview yielded up to 39 symptom-level evaluations. Eight symptom tags were excluded because of incomplete or corrupted audio recordings that precluded reliable transcription. The final dataset comprised 3931 symptom evaluations. Demographic characteristics of participants are shown in Table 2. The mean age of participants was 52.8 (SD 9.2) years, and the majority (n=88, 87.1%) were female participants. The most common primary cancer was breast cancer (n=70, 69.3%). Most participants (n=85, 84.2%) were receiving treatment after initial diagnosis or recurrence, while a smaller proportion were in posttreatment follow-up, remission follow-up, palliative care, or had unknown treatment status. Major depressive episodes were identified in 17.8% (n=18) of the participants. The distribution of clinician-rated symptom scores was skewed toward lower severity.

Across all 3 models, the concordance with board-certified psychiatrists on the 0 to 3 ordinal scale was high (Table 3 and Figure S1 in Multimedia Appendix 3). GPT-4o showed the highest agreement (ICC=0.872), followed by Claude 3.5 (ICC=0.838) and Gemini 2.5 (ICC=0.816).

When scores were dichotomized at ≥2 to indicate clinically significant items, all models demonstrated high performance (Table 3 and Figure 1). GPT-4o showed the highest F₁-score (0.837), followed by Claude 3.5 (0.813) and Gemini 2.5 (0.775). In the sensitivity analysis excluding ambiguous mismatches, F₁-scores were numerically higher for all 3 models, while the model ordering remained unchanged (Table S1 in Multimedia Appendix 3).

Directional error patterns for binary classifications (score ≥2 vs <2) are summarized in Figure 1. Across all models, overestimation errors (LLMs: ≥2, clinicians: <2) were more frequent than underestimation errors (LLMs: <2, clinicians: ≥2), but the relative proportions differed by model. Overestimation accounted for 142 of 248 (57.3%) errors for GPT-4o, 211 of 301 (70.1%) errors for Claude 3.5, and 225 of 355 (63.4%) errors for Gemini 2.5 (Figure 1).

Paired Wilcoxon signed-rank tests comparing the total symptom burden between each LLM and the clinician reference standard showed no statistically significant difference for GPT-4o (adjusted P=.15), whereas Claude 3.5 (adjusted P<.001) and Gemini 2.5 (adjusted P=.002) yielded higher total burdens than the clinicians (Table 3).

All binary mismatches between LLM and clinician ratings were classified by the primary adjudicator using the consensus-developed error taxonomy (Table 1), and the resulting classification frequencies are summarized in Table 4. Ambiguous cases accounted for a substantial proportion of disagreements across all models, comprising 88 of 301 (29.2%) mismatches for Claude 3.5, 118 of 355 (33.2%) mismatches for Gemini 2.5, and 97 of 248 (39.1%) mismatches for GPT-4o. A global difference across models was detected (Wald χ²_₂=6.58; P=.04). In the Holm-adjusted pairwise comparisons, GPT-4o had higher odds of ambiguity than Claude 3.5 (adjusted P=.03), whereas the other model contrasts were not significant (Table S3 in Multimedia Appendix 3). Neither the clinical-domain effect nor the model-by-domain interaction was statistically significant. No patient-characteristic main effect remained significant after Holm adjustment, and no model interaction was detected for age, clinician-rated item burden, major depressive episode status, or sex (all interaction-adjusted P>.99; Table S4 in Multimedia Appendix 3).

Among all model-specific mismatches, nonambiguous overestimation and underestimation accounted for 145 of 301 (48.2%) and 68 of 301 (22.6%), respectively, for Claude 3.5; 141 of 355 (39.7%) and 96 of 355 (27.0%) for Gemini 2.5; and 73 of 248 (29.4%) and 78 of 248 (31.5%) for GPT-4o (Table 4). Thus, even after excluding ambiguous cases, overestimation errors remained relatively more frequent than underestimation errors for Claude 3.5 and Gemini 2.5, whereas GPT-4o showed a more balanced distribution.

Figure 2 shows the composition of nonambiguous errors by subtype, displaying the model-wise distributions after renormalizing the proportions within each model and excluding ambiguous cases. Criteria-related errors were the most frequent nonambiguous subtype across all 3 models. Claude 3.5 and Gemini 2.5 predominantly overestimated severity based on misapplied criteria, whereas GPT-4o showed a more balanced profile with relatively higher criteria-related underestimation. Implicit cue errors accounted for smaller proportions in GPT-4o than in Claude 3.5 and Gemini 2.5. Less frequent subtypes, such as errors involving other information, somatic confounding, temporal misalignment, comprehension problems, and personality attribution, each accounted for only a small proportion of nonambiguous mismatches in all models.

Meta-evaluation scores differed significantly between the models across all 4 dimensions (citation, structure, mapping, expansion; all P<.001; Table 5). Claude 3.5 had the highest citation and structure scores, Gemini 2.5 had intermediate scores, and GPT-4o had the lowest scores on these dimensions. Mapping scores were similar for Claude 3.5 and Gemini 2.5 and lower for GPT-4o, whereas expansion scores were similar for GPT-4o and Claude 3.5 and lower for Gemini 2.5 (Table 5).

In exploratory domain-adjusted cross-classified mixed-effects models, mapping was inversely associated with absolute error, whereas structure and expansion were positively associated with error across all 3 source models. Citation was positively associated with error for GPT-4o and Claude 3.5 but not for Gemini 2.5 (Table S5 in Multimedia Appendix 3). In the human-reference sensitivity analyses, coefficient directions were consistent with those observed in the primary analysis for all 4 dimensions, although statistical significance varied across dimensions and source models (Figure S2 in Multimedia Appendix 4).