This was a retrospective study approved by the Institutional Review Board at the National Institutes of Health (NIH) Clinical Center (number 000037). Informed consent was waived. All patient notes used in this study were deidentified before use. Participants received no compensation because the study involved retrospective analysis of existing data without direct participant involvement.

In this retrospective study approved by the institutional review board, we used 2 datasets from 2 medical institutions, the NIH Clinical Center (center 1) in Bethesda, United States, and the Beth Israel Deaconess Medical Center (center 2) in Boston, United States. The first dataset (referred to as NIH, henceforth) comprised 180 history and physical examination notes corresponding to the first visits of 180 participants randomly chosen from patients who were admitted and underwent computed tomography scans at center 1 between 2000 and 2023. The second dataset (Medical Information Mart for Intensive Care [MIMIC]), a subset of the publicly available MIMIC-IV-Note dataset [28,29], consisted of 200 discharge notes corresponding to 200 participants randomly chosen from patients admitted to the hospital and emergency department at center 2. The notes were deidentified before use with the Stanford deidentifier tool and reviewed by authors MN and VB [30].

Each note was labeled for the presence or absence of 29 conditions, a simplified version of the 30 conditions covered by the Elixhauser Comorbidity Index (ECI) [31]. The 2 comorbidities of “diabetes, uncomplicated” and “diabetes, complicated” in the ECI are merged to a single category called “diabetes” (the full list is available in Table 1). The ECI was used as a framework for selecting a broad and clinically diverse set of conditions that may be documented in admission and discharge notes and may be relevant targets for automated clinical information extraction. The annotation task was not intended to reproduce formal Elixhauser coding, calculate an ECI score, or assign billing codes. Rather, annotators determined whether each condition was present or absent in the narrative documentation. The NIH notes were manually labeled by a medical student (MN), while the MIMIC notes were labeled by a first-year radiology resident (VB), both under the supervision of a senior radiologist (RMS) with more than 30 years of experience. These binary “yes/no” manual labels were used as the reference standard in the subsequent analysis of performance of AI models.

Both datasets were combined, and each note was labeled by each of 46 LLMs, including 4 closed-source models: GPT-4, GPT-4o, Claude 3.5 Sonnet (version 20240620), and Claude 3.5 Sonnet (version 20241022), 13 Llama models (spanning versions 2, 3, 3.1, 3.2, and 3.3), and 29 Qwen models (spanning versions 1, 1.5, 2, and 2.5). Table S1 in Multimedia Appendix 1 provides a full list of the models. Figure S1 in Multimedia Appendix 1 shows an overview of the prompting strategy. The same prompt structure (Figure S1 in Multimedia Appendix 1), asking the model if the patient had a given comorbidity based on the note, was used for every model. An individual prompt was used for each diagnosis and each note; thus, each response corresponded to a unique instance of the tuple (model, note, and diagnosis). If the model refused to answer (due to false-positive content filters in the models) or answered with empty strings, these responses were considered missing and dropped in subsequent analysis. The outputs from the model were also free text, and though the prompt instructed the model to end its answer with yes or no, not all models adhered to the instruction. Therefore, the GPT-o3-mini-high reasoning model was used to analyze the model outputs and parse the answer to a binary yes/no. If the model refused to answer, these were also considered to be missing responses. The prompt used for this step is also reported in Multimedia Appendix 1.

The measures used to evaluate performance were recall, precision, and the F₁-score. The metrics and their CIs were evaluated for every model and both datasets. The 95% CIs for precision and recall were computed using the exact Clopper-Pearson method [32]; for the F₁-score, 95% CIs were obtained via a nonparametric bootstrap with 2000 resamples of notes (not individual note-diagnosis pairs, to preserve within-note correlation across diagnoses), recalculating precision, recall, and F₁-score on each resampled dataset and taking the 2.5th and 97.5th percentiles of the resulting F₁-score distribution as the interval bounds.

Additionally, we trained generalized linear models (GLMs) with a binomial family and logit link to predict the recall and precision (separately) using the covariates: model family (Llama, Qwen, GPT, and Claude), log10(model_size) (used for open-weights models only), length of the note (number of words), and the model recency (number of days since the model was released). The full model coefficients are provided in Tables S2 to S5 in Multimedia Appendix 1. The recall model was trained to predict the model response (true/false) on the subset of instances (model, note, and diagnosis), on which the reference standard label was true. Likewise, the precision model was trained to predict the reference standard label on the subset of instances for which the model prediction was true. For the F₁-score, 95% CIs were obtained using the same note-level bootstrap procedure described above, applied to the GLM-derived precision and recall estimates. This was done primarily to analyze the trends of performance for the open-weights models along 2 axes: length of time since the model was released and the model size. This also enabled statistically grounded comparisons of performance across LLM model families while accounting for variability in model size and generation. Additional GLMs (for precision and recall), which also incorporated the dataset and the comorbidities as predictive factors, were trained to explore differences across datasets and diagnoses; for each model family and diagnosis, estimated marginal means of recall and precision were obtained and combined into an F₁-score, with a 95% CI from the same note-level bootstrap procedure described above (Figure S2 and Tables S6 and S7 in Multimedia Appendix 1). Mixed-effects models with random intercepts for the note and the diagnosis were trained to account for the repeated-measures structure (results and Tables S8 and S9 in Multimedia Appendix 1). Because the open-weights models include both text-only and vision-language variants, we additionally tested architecture type (multimodal vs text-only) as a covariate in the recall and precision GLMs to assess whether it confounds the model-family, size, and recency effects. As a check on the fidelity of the quoted evidence supporting each model’s response, we additionally verified whether the quoted text excerpts in each response were actually present in the corresponding note, using an automated string-matching procedure (Table S12 in Multimedia Appendix 1). As a check on the reliability of the GPT-o3-mini-high parsing step, we additionally compared its binary judgments against 3 alternate judge models (GPT-4, GPT-4o, and Claude 3.5 Sonnet) and found high agreement (Table S13 in Multimedia Appendix 1). The analysis was done using R (version 4.3.3; R Foundation for Statistical Computing) and used the emmeans (version 1.10.1; Comprehensive R Archive Network) and dplyr (version 1.1.4; Posit Software, PBC) packages.

All open-weights models, except Llama 3.1 405B, were run on a high-performance computing cluster (Biowulf at the NIH) using the vLLM (version 0.7.3; PyTorch Foundation) package in Python (version 3.12.9; Python Software Foundation). The Llama 3.1 405B and Claude models were run using the Amazon Web Services Bedrock platform. The GPT models were run using the Azure OpenAI Service. The temperature hyperparameter was set to 0 while running the models. The max_tokens parameter, representing the maximum number of tokens to generate per output sequence, was set to 8192 to allow long model outputs.

A total of 380 notes, 180 from the NIH and 200 from the MIMIC dataset, were used in this study. The median lengths of a note in the NIH and MIMIC datasets were 1043 (IQR 670-1543) and 1553 (IQR 1106-1948) words, respectively. The prevalence of each Elixhauser comorbidity in each dataset in the reference standard is reported in Table 1.

In this study, we benchmarked the performance of 46 LLMs, including 4 closed-source commercial models (GPT-4, GPT-4o, Claude 3.5 Sonnet [version 20240620] and Claude 3.5 Sonnet [version 20241022]), and 42 open-weights LLMs from 2 popular model families, Llama and Qwen, spanning various model sizes and generations. Using GLMs, we also analyzed how the performance varied based on various factors such as the model size, family, and generation and the length of the notes. We found that in the labeling task, larger open-weights models such as the Llama 3.3 70B and the Qwen 2 72B performed at par or better than the commercial models in terms of F₁-scores. Controlling for model factors such as size and recency, the Qwen family had higher precision and F₁-scores than the Llama family of models. Model size primarily improved precision (recall showed much smaller improvement), leading to an overall increase in F₁-scores. Recent open-weights models have lower recall but significantly higher precision than older models, resulting in an improvement in F₁-scores. Perhaps surprisingly, longer notes seemed to improve the performance of open-weights models (the closed-source models had a relatively stable performance) in terms of both recall and precision.

The performance of LLMs on similar tasks has been explored in prior literature. For example, several papers have studied the task of note summarization and found that LLMs can perform on par with clinicians, with low hallucination and omission rates [33,34]. Other papers have tackled the task of labeling clinical notes; for example, Burford et al [20] evaluated the performance of GPT-4 in identifying helmet status of patients injured in micromobility-related accidents from clinical notes, evaluating the extraction of presence of a single condition. Wu et al [7] extracted a variety of conditions from clinical notes but did not report performance for individual conditions, likely due to the rarity of the target diseases. McInerney et al [6] extracted 10 common conditions but relied on a single, now-outdated LLM, achieving an average F₁-score of approximately 0.42. Other papers have focused on labeling other forms of clinical text such as radiology or pathology reports [4,35]; however, clinical notes considered in this study are usually longer and pose greater challenges than radiology and pathology reports. Most existing papers focus on single LLMs or single conditions, unlike this current study that uses a multi-institutional set of notes, a diverse set of comorbidities to be labeled, and evaluates 46 models, including 2 families of closed-source commercial LLMs (GPT and Claude) and 2 families of open-weights LLMs (Llama and Qwen). Additionally, we modeled the performance of these LLMs using factors such as model family, size, and recency to provide insight into how these factors affect performance.

While it is somewhat dogmatic to assume that newer and larger model sizes will necessarily be better at every task, it may not always be the case. For example, Zhou et al [36] found that larger and more instructable models may become less reliable. Our results show the general trend of improving performance with increasing model size holds for the note-labeling task. However, outliers exist, and, in some cases, the general trend may not be true; for example, the recall of Llama 3.1 405B is less than that of the Llama 3 8B model (75.8, 95% CI 73.7-77.9 vs 83.8, 95% CI 81.9-85.6). When considering model recency, we found that the recall shows a decreasing trend—recent models have lower recall than older models (though this was accompanied by an improvement in precision and an improvement in the overall F₁-score). Notably, within the Qwen family, Qwen 2.5 72B showed lower recall than Qwen 2 72B (58.4% vs 68.5%), offset by higher precision (80.9% vs 76.7%), reducing the overall F₁-score (67.9% vs 72.4%) in the newer generation model.

One of the surprising findings in the study was that the open-weights LLMs tended to perform better (both precision and recall) on longer notes when controlling for model factors such as family, size, and recency. The effect did not go away when controlling for the dataset (Tables S6 and S7 in Multimedia Appendix 1), and in fact, the dataset was not a significant factor when modeling precision for the open-weights models. This may perhaps be due to more redundancy or more explicit presentation of information in longer notes.

In this study, we chose to combine the datasets from 2 different institutions (n=180 and n=200) to create a single large cohort (n=380) on which all the models were tested. This increased the diversity of the notes, both note type and institution (history and physical notes from the NIH and discharge notes from the MIMIC datasets), as well as the size of the cohort, giving us more robust estimates of real-world performance and trends. We also did not use the dataset or note type as factors in the GLMs. This simplicity allowed us to focus on the model factors such as family, size, and recency and analyze how those influence performance. Additionally, an alternative formulation of the GLMs with dataset (consequently the note type, as well) for the open-weights models shows that the coefficients (and trends) of the fitted models are very similar, particularly for the model-related factors (note length was significantly associated with the dataset; Tables S6 and S7 in Multimedia Appendix 1). This also highlights the robustness of the GLM-based analysis method used here, a conclusion further supported by a mixed-effects model sensitivity analysis accounting for repeated measures within notes and diagnoses (Tables S8 and S9 in Multimedia Appendix 1).

Our study has several limitations. First, the reference standard was established by a medical student and a first-year radiology resident under the supervision of a senior radiologist, rather than by internal medicine physicians or certified medical coders. Although the annotation task involved simplified binary identification of clinical conditions in narrative notes rather than formal Elixhauser coding or billing-code assignment, some conditions may have required greater specialty-specific or coding expertise. In addition, the annotations were completed in sequential phases without a double-labeled subset, precluding formal assessment of interrater reliability. The senior radiologist provided guidance and spot-checked the annotations, but this quality-control process does not substitute for specialist adjudication or quantitative measurement of agreement. Consequently, some degree of reference-standard misclassification cannot be excluded. Second, the GLMs used to model trends in model family, size, and recency (Figures 2 and 3) exclude comorbidity as a predictor, in favor of the interpretability of those specific coefficients. A separate analysis incorporating comorbidity (Figure S2 in Multimedia Appendix 1) confirmed substantial variation in performance across conditions, underscoring the importance of evaluating LLMs on the specific clinical use case before deployment. Third, since the MIMIC-IV-Note dataset is publicly available, it may have been included, in part or whole, in the training corpora of at least some of the LLMs we evaluated, and this may have affected the reported performance. However, including the dataset as an additional covariate in the GLMs for precision and recall did not alter the coefficients of model family, model size, and model recency (note length showed more modest shifts), suggesting that the principal associations identified in our analyses were robust to adjustment for systematic differences between the public MIMIC-IV-Note dataset and the nonpublic institutional dataset. Fourth, we used a fixed prompt for all our models and did not evaluate the variability of results to variation in the prompts. Additionally, the prompt did not explicitly instruct models to distinguish between comorbidities active at the time of admission and those noted only as part of the patient’s past history, and models may not have reliably made this distinction on their own. Fifth, our automated parsing of free-text responses into binary judgments, while shown to have high agreement across independent judge models (Table S13 in Multimedia Appendix 1), was not manually verified at scale and cannot rule out an error shared by all LLM judges—for example, a note phrasing that misleads every judge model in the same way. Finally, we dropped missing/unparseable responses (3.99% overall) from our analysis. The median model had fewer than 0.4% missing responses, but 4 models—Qwen 1 14B (65.0%) and the 3 Llama 2 variants (30.7%‐31.0%)—had substantially higher rates. A sensitivity analysis recomputing performance with missing responses treated as incorrect shows that this had negligible effect for nearly all models, but shifted F₁-score by 10 to 53 percentage points for these 4 high-missingness models. We therefore expect our reported performance to be reliable for most models, with the results for these 4 specifically interpreted with this caveat in mind.

We performed a comprehensive evaluation of 2 families of closed-source commercial LLMs (GPT and Claude) and 2 families of popular open-weights LLMs (Llama and Qwen) on the task of labeling clinical notes. We also analyzed the trends of performance with respect to various factors such as model family, model size, and model recency. Our results present a comprehensive picture of the LLM performance on the clinically relevant task of labeling clinical notes.