The overall study design is illustrated in Figure 1. This was a retrospective framework development and internal validation study based on EMR data from a single tertiary liver disease center. Eligible patients were identified from the EMR system, and structured laboratory indicators and unstructured clinical narratives were extracted. LLM-based semantic embeddings were generated from standardized clinical text inputs and integrated with clinical laboratory variables. Classification models were then developed and evaluated against ML-only pipelines and direct LLM inference.

We analyzed patient data retrieved from the EMR system of Beijing Youan Hospital, Capital Medical University. The initial screening included 162,159 patients seen between January 2010 and June 2025. The EMR contained structured laboratory test results and unstructured clinical narratives generated during routine clinical care, including imaging, endoscopy, and pathology-related reports, if available.

The study focused on patients diagnosed with CHB, DILI, and AILD. Within the AILD spectrum, AIH and PBC were included as the major subtypes. Patients with rarer autoimmune hepatobiliary diseases, including primary sclerosing cholangitis and IgG4-related sclerosing cholangitis, were excluded from the final analytical cohort because their limited sample sizes precluded robust statistical modeling.

Inclusion criteria were (1) age ≥18 years, (2) at least 2 independent laboratory records supporting the same diagnosis, and (3) availability of medication and treatment records. Diagnostic labels were assigned according to established international guidelines supplemented by Chinese national guidelines. Specifically, AIH was defined according to the International Autoimmune Hepatitis Group simplified criteria [23], PBC according to the European Association for the Study of the Liver and Chinese Society of Hepatology guidelines [24,25], DILI according to the Roussel Uclaf Causality Assessment Method [26,27], and CHB according to the American Association for the Study of Liver Diseases along with the 2022 guidelines jointly issued by the Chinese Society of Hepatology and the Chinese Society of Infectious Diseases [28,29].

Because the enrollment period spanned from 2010 to 2025, all historical cases were retrospectively readjudicated by a senior hepatologist using archived clinical, serological, laboratory, imaging, and treatment data. Cases that no longer met the current criteria or lacked sufficient evidence for diagnostic confirmation were excluded to ensure label consistency across the 15-year study window.

Exclusion criteria were (1) coexistence of multiple competing liver etiologies, such as AIH-PBC overlap syndrome or CHB combined with DILI; (2) major comorbidities likely to independently affect liver biochemistry, lipid metabolism, or imaging findings, including malignant tumors and diabetes mellitus; (3) alcohol-related liver disease; (4) duplicate records for which only the first eligible encounter with a confirmed diagnosis was retained; and (5) patient-level missingness exceeding 40% across candidate clinical variables. These criteria were applied to reduce etiological ambiguity and ensure diagnostic certainty in this initial multiclass classification task involving nonoverlapping target etiologies. Patients with common metabolic comorbidities other than diabetes mellitus, including hypertension, dyslipidemia, obesity, and metabolic dysfunction–associated steatotic liver disease with no diabetes, were retained in the cohort to preserve generalizability to typical clinical populations.

After applying the inclusion and exclusion criteria, the final cohort comprised 7543 patients across 3 disease categories, AILD (n=702), DILI (n=992), and CHB (n=5849). The AILD group consisted of AIH (n=256) and PBC (n=446). The cohort was stratified by the 4-class labels (AIH, PBC, DILI, and CHB) and randomly split into training (80%) and internal holdout validation (20%) sets. This stratification ensured class proportion preservation for both the 3-class task (where AIH and PBC are merged into AILD) and the 4-class task. Within the training set, we performed 5-fold cross-validation for dimensionality reduction and hyperparameter tuning [30]. The internal holdout validation set was kept separate from all model development procedures and was used only for final performance evaluation [31].

The study was approved by the Ethics Committee of Beijing Youan Hospital, Capital Medical University under protocol LL-2025‐029-K. The approved retrospective data use protocol covered the extraction and analysis of deidentified EMR data collected between January 2010 and June 2025. The requirement for informed consent was waived by the ethics committee. Data privacy was protected by deidentifying all records before analysis. All LLM encoding and reasoning tasks were performed locally to minimize the risk of information leakage.

Three 8B-parameter LLMs were selected as clinical data encoders, including Huatuo-o1 [32], II-Medical [33], and Qwen3 [34]. Huatuo-o1 and II-Medical are domain-specific medical LLMs, while Qwen3 is a general-purpose LLM. Exact repository identifiers and checkpoint versions for local deployment of each model are provided in Multimedia Appendix 1 to support computational reproducibility. To determine whether large medical LLMs provided incremental value over lighter text encoders, we also evaluated 2 lightweight natural language processing encoders, Doc2Vec [35] and BGE-M3 [36].

For each patient, structured and unstructured EMR contents were consolidated into a standardized modular text format. Examination items were organized as “<examination item> : <content>” and grouped into clinical modules, including demographics, laboratory tests, imaging reports, endoscopy reports, and pathology-related descriptions when available. To reduce potential label leakage before LLM encoding, explicit diagnostic labels, target disease names, discharge diagnoses, prescribed medication names, and other postdiagnostic information were removed or masked using predefined filtering rules [37]. These rules were applied to both structured medication fields and unstructured free-text narratives to identify and mask medication names and disease-specific treatment keywords, including highly diagnosis-informative terms such as specific antiviral agents and ursodeoxycholic acid. The complete list of high-risk keywords used for filtering is provided in Multimedia Appendix 2. A processed example is shown in Multimedia Appendix 3.

For each patient, dense embeddings of dimension 4096 were extracted by mean-pooling the token-level hidden states of the final transformer layer of each LLM, followed by L2 normalization, in a zero-shot setting, without any task-specific fine-tuning. To balance computational efficiency with information retention, principal component analysis (PCA) was applied to project the embeddings into lower-dimensional representations. Three target dimensions, 32, 64, and 128, were systematically evaluated. PCA fitting was performed only within the training data during cross-validation to prevent information leakage. The resulting LLM embeddings were then integrated with clinical laboratory variables to construct LLM-integrated feature sets. This integration strategy preserved interpretable clinical measurements while adding latent semantic representations derived from unstructured clinical narratives.

To explore the direct clinical reasoning capability of LLMs, we implemented a structured prompt engineering workflow. Models were prompted via a structured template that specified the role definition, task instructions, input constraints, and standardized output formats for disease classification and medication recommendation [38,39]. The prompt template is shown in Multimedia Appendix 4. All reasoning tasks were conducted in a zero-shot setting without task-specific fine-tuning [40].

To improve reproducibility, deterministic generation was used whenever supported by the model. The temperature was set to 0, and sampling was disabled by setting do_sample to false. Detailed inference hyperparameters, including maximum output length and decoding settings for each model, are provided in Multimedia Appendix 5. All LLM inference, including embedding extraction and zero-shot reasoning, was performed locally on a workstation equipped with 8 NVIDIA GeForce RTX 2080 Ti graphics processing units with 11 GB of video random access memory each.

Direct LLM diagnostic performance was evaluated by comparing the model-predicted disease category, parsed from the structured output field specified in the prompt template, with the reference diagnosis. Outputs that failed to match any of the predefined categories were classified as incorrect predictions.

Medication recommendation performance was assessed as an exploratory end point. Prescription records served as the reference list for comparison. To prevent label leakage, all medication-related content was removed from the input before it was supplied to the LLMs, and the models were prompted to generate medication recommendations from the remaining clinical narratives and laboratory data.

A medication match was defined as an exact or synonym-normalized match between a recommended drug class and a recorded medication class [41]. For each patient, medication precision was defined as the number of correctly recommended drug classes divided by the total number of recommended drug classes, and medication recall was defined as the number of correctly recommended drug classes divided by the total number of prescribed drug classes. Patient-level macroaveraged precision, recall, and F₁-score were averaged across patients and reported as the primary medication recommendation metrics. When no medication was recommended, precision and F₁-score were set to 0 for that patient. The any-match rate was retained only as a secondary descriptive indicator. Because actual prescriptions may reflect disease severity, contraindications, physician preference, drug availability, and temporal changes in treatment strategy, medication matching was interpreted as an exploratory indicator of clinical plausibility rather than a measure of treatment correctness.

For comparative analysis, an ML-only pipeline was developed using structured clinical laboratory variables without LLM-derived embeddings [42]. To ensure a fair comparison with the LLM-integrated ML models, the ML-only model and the LLM-integrated ML models used the same structured clinical variable set and the same preprocessing pipeline. The only difference was that the LLM-integrated models additionally incorporated LLM-derived semantic embeddings.

For continuous variables, only those with at least 80% of nonmissing values (ie, <20% missingness) were retained, and missing values were imputed using the k-nearest neighbors algorithm, which estimates missing values based on intersample similarity among clinical variables. For categorical variables, categorical missingness itself may carry clinical information (eg, a test not being ordered); missing values were therefore encoded as a separate missing category rather than imputed to retain potentially informative missingness patterns in real-world EMRs. This missingness-handling scheme was applied specifically to the variable set used for model development and differs from the descriptive statistics reported, which summarize the originally observed (nonimputed) values.

All retained clinical variables were used directly for modeling without additional feature selection or dimensionality reduction. To prevent information leakage, imputation and scaling were fitted only on the training subset within each cross-validation fold and were then applied to the corresponding validation subset [43]. After hyperparameter selection, the complete preprocessing and modeling pipeline was refitted on the full development set and applied once to the internal holdout validation set for final performance evaluation.

Two classification tasks were implemented. The 3-class task classified patients as AILD, DILI, or CHB. The 4-class task classified patients as AIH, PBC, DILI, or CHB to evaluate fine-grained discrimination within AILD. Models were developed within the development set using stratified 5-fold cross-validation. Five downstream ML algorithms were evaluated in parallel, including random forest [44], multilayer perceptron (MLP) [45], logistic regression (LR) [46], Extreme Gradient Boosting (XGBoost) [47], and support vector machine (SVM) [48]. For each candidate hyperparameter configuration, model training and validation were performed across all 5-folds, and the mean validation macro F₁-score was computed. The configuration with the highest mean validation macro F₁-score was selected for final deployment. The hyperparameter search space and fixed training parameters are provided in Multimedia Appendix 6.

Class imbalance was addressed during model training. For random forest, LR, and SVM, class_weight = 'balanced' was used to assign weights inversely proportional to class frequencies. For MLP and XGBoost, equivalent sample-weighting strategies were applied during training. These strategies were used only within the training folds [49].

All preprocessing steps were incorporated into fold-specific training pipelines. For models requiring feature scaling, including MLP, LR, and SVM, z score normalization was fitted only on the training subset and then applied to the validation subset. PCA for LLM embeddings followed the same rule and was fitted only on the training subset within each fold. After cross-validation, the selected preprocessing steps and hyperparameter configuration were refitted on the full development set and evaluated once on the internal holdout validation set.

Model performance on the internal holdout validation set was assessed using accuracy, macro precision, macro recall, macro F₁-score, and macroaveraged area under the receiver operating characteristic curve. Macroaveraged metrics were used to assign equal weight to each disease class and reduce bias caused by the predominance of CHB cases. All performance metrics are reported with 95% CIs computed by stratified bootstrap resampling with 1000 iterations on the internal holdout validation set, preserving class proportions across iterations. The ML algorithm achieving the highest cross-validation performance within the training set was selected as the representative model and subsequently evaluated on the internal holdout validation set for cross-strategy comparison.

As a sensitivity analysis, we also performed temporal internal validation using cases from 2010 to 2019 for training and cases from 2020 to 2025 for testing. Hyperparameter tuning was conducted only within the 2010‐2019 training set using 5-fold cross-validation, and the 2020‐2025 temporal test set was evaluated once.

Normality of continuous variables was assessed once in the pooled cohort using D’Agostino and Pearson omnibus test. Variables with fewer than 20 available observations were treated as non–normally distributed by default, given insufficient power to assess normality. Regardless of distribution, continuous variables were summarized as median (IQR) for the overall cohort and for each of the 4 diagnostic groups (AIH, PBC, DILI, and CHB), together with the number of patients with an available result in that specific column (N); group comparisons used 1-way ANOVA for variables identified as normally distributed in the pooled cohort and the Kruskal-Wallis test otherwise, and were not reported (shown as "—") when any of the 4 groups had fewer than 2 valid observations. Categorical variables were summarized as n/N (%) for the overall cohort and for each diagnostic group, where N is the number of patients with an available test result in that specific column rather than the corresponding column’s full sample size, since not all laboratory and serological tests were obtained in every patient; group comparisons used the chi-square test and were not reported when any expected cell frequency was below 5. For each variable, the number and percentage of patients without an available result in the full cohort (missing, n [%]) were additionally reported, using the full cohort size as the denominator. All statistical tests were 2-sided, and P<.05 was considered statistically significant. Statistical analyses were performed in Python (version 3.8; Python Software Foundation) using SciPy (version 1.7.3; SciPy Developers).

Baseline demographic and key clinical laboratory characteristics of the study cohort are summarized in Table 1. The internal holdout validation set exhibited distributions broadly comparable with those of the overall cohort, supporting its use for internal validation. Among the 7543 included patients, demographic features, biochemical profiles, viral hepatitis markers, autoimmune serological markers, and urinalysis findings varied across the 4 disease groups.

Patients with PBC and AIH were older and predominantly female, with median ages of 58.0 and 54.0 years and male proportions of 14.3% and 13.7%, respectively. In contrast, patients with CHB were younger and predominantly male, with a median age of 42.0 years and a male proportion of 59.5%. Biochemical profiles varied in patterns consistent with the corresponding disease categories. Cholestatic markers were highest in patients with PBC, with median γ-glutamyl transferase and alkaline phosphatase values of 154.2 U/L and 169.0 U/L, respectively. Hepatocellular injury was most prominent in patients with DILI, with a median alanine aminotransferase of 106.3 U/L and the highest urinary bilirubin positivity rate (37.2%). Globulin levels were highest in patients with PBC (median 35.1 g/L), accompanied by lower estimated glomerular filtration rate values in both PBC and AIH compared with CHB, suggesting a degree of systemic involvement in autoimmune disease groups. Antimitochondrial antibody (AMA) positivity was frequent in PBC (81.6%), and antinuclear antibody (ANA) positivity was prevalent in both AIH and PBC (96.8% and 86.3%, respectively). Hepatitis B virus–related markers were concentrated in CHB, including hepatitis B surface antigen positivity of 91.9% and hepatitis B e antigen positivity of 40.3%. These serological and biochemical patterns were consistent with established diagnostic patterns [50-54] and supported the validity of the cohort for downstream modeling. Complete statistical results for all variables are provided in Multimedia Appendix 7.

To evaluate the discriminative potential of LLM-embedding models, we first conducted a 3-class task (AILD vs DILI vs CHB). As a reference, the ML-only model was developed using the full set of retained structured clinical laboratory variables without LLM-derived embeddings. This model achieved an accuracy of 0.893 (95% CI 0.877‐0.908) and a macro F₁-score of 0.791 (95% CI 0.761‐0.820) on the internal holdout validation set.

We then evaluated LLM embedding-only models using LLM-derived semantic embeddings as stand-alone features. These models were trained without expert-defined text feature engineering or manually selected semantic variables. With a PCA dimensionality of 64, the Qwen3 embedding-only model achieved the highest accuracy of 0.919 (95% CI 0.907‐0.932), exceeding the ML-only model. The Huatuo-o1 and II-Medical embedding-only models achieved comparable performance, indicating that LLM-derived embeddings alone retained clinically relevant discriminative information from heterogeneous EMR inputs (Multimedia Appendix 8).

To contextualize the contribution of LLM-derived semantic representations, we compared their embedding-only models with 2 lightweight text representation methods. The 3 LLM embedding-only models outperformed Doc2Vec, which achieved an accuracy of 0.836 (95% CI 0.818‐0.854) and a macro F₁-score of 0.635 (95% CI 0.596‐0.672), and BGE-M3, which achieved an accuracy of 0.838 (95% CI 0.819‐0.856) and a macro F₁-score of 0.631 (95% CI 0.591‐0.669), on the internal holdout validation set (Multimedia Appendix 9). These findings suggest that LLM-derived embeddings provided more discriminative semantic information than the lightweight text encoders evaluated in this study.

Taken together, embedding-only analysis showed that LLM-derived representations could encode clinically relevant information from standardized EMR inputs and provide competitive performance in broad etiological liver disease classification. These results support their use as high-dimensional feature representations for downstream ML classification.

We next evaluated whether LLM-derived embeddings provided complementary information when combined with structured clinical laboratory variables. After feature integration, the LLM-integrated ML models achieved accuracies ranging from 0.925 to 0.929 on the internal holdout validation set, corresponding to improvements of 0.8-2.8 percentage points over the respective LLM embedding-only models (Figures 2A-2C). All 3 LLM-integrated ML models also outperformed the ML-only baseline, with accuracy increasing by 3.2-3.6 percentage points and macro F₁-score increasing by 4.4-4.6 percentage points (Table 2). Although the II-Medical–integrated ML model achieved the highest accuracy of 0.929 (95% CI 0.916‐0.941), the 3 LLM-integrated models showed comparable overall performance, with overlapping 95% CIs across the reported metrics.

Class-specific precision analysis showed that performance differences were more evident for AILD and DILI than for CHB (Table 3). AILD precision was 0.700 (95% CI 0.623‐0.777) in the ML-only model and ranged from 0.775 to 0.796 in the LLM-integrated ML models. DILI precision was 0.606 (95% CI 0.550‐0.661) in the ML-only model and ranged from 0.738 to 0.744 in the LLM-integrated ML models. In contrast, CHB precision remained high across all models, ranging from 0.977 to 0.991.

We further assessed the computational cost of LLM-integrated ML models. Per-patient end-to-end inference time, including tokenization, LLM embedding extraction, dimensionality reduction, feature concatenation, and classifier prediction, was longer than that of the ML-only model but remained approximately 670‐685 ms per case in the local benchmark (Multimedia Appendix 10). All models were deployed locally on graphics processing units, and the embedding-based pipeline used a single forward pass without autoregressive token decoding. This design explains the lower latency compared with API-based generative reasoning. These findings suggest that LLM-derived embeddings provide information complementary to structured clinical laboratory variables in the 3-class task. The contribution was most apparent for AILD and DILI, whereas CHB classification remained robust across modeling approaches.

We next evaluated the feature integration strategy in a more refined 4-class classification task (AIH vs PBC vs DILI vs CHB) to assess whether LLM embeddings could further discriminate between clinically related AILD subtypes. The ML-only model achieved an accuracy of 0.874 (95% CI 0.857‐0.889) and a macro F₁-score of 0.665 (95% CI 0.626‐0.701) on the internal holdout validation set. In comparison, the 3 LLM-integrated ML models achieved accuracies ranging from 0.920 to 0.922, corresponding to improvements of 4.6-4.8 percentage points over the ML-only model (Table 4). Macro F₁-scores also increased, ranging from 0.717 to 0.734 across the LLM-integrated models. The Huatuo-o1–integrated ML model achieved the highest macro F₁-score of 0.734 (95% CI 0.690‐0.774), while the 3 LLM-integrated ML models showed broadly comparable performance with overlapping 95% CIs across the reported metrics.

The LLM-integrated ML models also showed more balanced discrimination across disease classes, with higher macroaveraged area under the receiver operating characteristic curves than the ML-only model in the 4-class task (Multimedia Appendix 11). These findings suggest that LLM-derived embeddings provided complementary information for the refined 4-class task rather than improving performance only in the majority CHB category.

Feature importance analysis identified both conventional clinical variables and LLM-derived components among the most informative features. Autoantibody markers, including AMA and ANA, remained highly ranked, consistent with their established relevance in AILD classification [55]. Several LLM-derived principal components, including PCA_2, PCA_18, and PCA_8, were also among the top-ranked features (Figure 3A). These components represent orthogonal dimensions of the compressed embedding space and should be interpreted as semantic representation features rather than directly interpretable clinical variables.

Together, these findings support the complementary value of LLM-derived representations when integrated with structured clinical variables for fine-grained liver disease classification.

To assess model stability under temporal evaluation, we performed a sensitivity analysis using cases from 2010 to 2019 for training and cases from 2020 to 2025 for testing. Hyperparameter tuning was performed only within the 2010‐2019 training set, and the 2020‐2025 temporal test set was evaluated once.

In the 3-class task, the LLM-integrated ML models maintained higher macro F₁-scores than the ML-only baseline under temporal evaluation, with macro F₁-scores ranging from 0.852 to 0.858 compared with 0.799 for the ML-only model. In the 4-class task, the same pattern was observed, with macro F₁-scores ranging from 0.743 to 0.752 for the LLM-integrated ML models compared with 0.702 for the ML-only model. These findings indicate that the relative advantage of LLM-derived embeddings over the ML-only baseline was preserved under temporal internal validation. Full temporal split results are provided in Multimedia Appendix 12.

Beyond the embedding-based integration framework, we examined whether LLMs could directly generate disease classification and medication recommendations through structured prompts. In direct diagnostic inference, Qwen3 achieved the highest performance among the 3 evaluated LLMs, with an accuracy of 0.721 and a macro F₁-score of 0.477. However, this performance remained lower than that of the LLM-integrated ML framework. For example, the Huatuo-o1–integrated ML model achieved an accuracy of 0.922 and a macro F₁-score of 0.734 in the 4-class task (Figure 3B).

Medication recommendation was evaluated as an exploratory end point by comparing model-generated drug classes with recorded prescriptions after medication name normalization. Agreement with actual prescriptions varied across models. II-Medical achieved the highest any-match rate of 57.5%, followed by Qwen3 at 52.6% and Huatuo-o1 at 18.9% (Multimedia Appendix 13). The same pattern was observed using list-level metrics. Patient-level macro precision was 0.542 for II-Medical, 0.493 for Qwen3, and 0.184 for Huatuo-o1. Patient-level macro F₁-score was 0.280 for II-Medical, 0.249 for Qwen3, and 0.088 for Huatuo-o1.

The low recommendation rate observed for Huatuo-o1 was partly related to its tendency to recommend observation or no specific treatment for a substantial proportion of patients with CHB. This may represent clinically appropriate judgment but results in low overlap with recorded prescriptions under the current evaluation framework.

Overall, direct zero-shot LLM reasoning showed limited stand-alone diagnostic and medication recommendation performance compared with the embedding-based integration framework. These findings support the use of LLMs primarily as feature encoders within a controlled ML pipeline rather than as stand-alone diagnostic or therapeutic decision systems.

In this single-center retrospective study, we developed an LLM-integrated framework for multiclass liver disease classification using heterogeneous EMR data. The framework combined semantic embeddings derived from free-text clinical reports and standardized clinical observations with structured laboratory variables. On the internal holdout validation set, the integrated models achieved higher performance than the ML-only model in both the 3-class task and the more granular 4-class task. The improvement was most evident for non-CHB categories and for fine-grained classification involving AIH and PBC, 2 AILD subtypes with overlapping biochemical profiles but different clinical management pathways.

A central finding of this study is that LLM-derived embeddings may provide information complementary to conventional laboratory variables. Clinical laboratory markers remain essential for liver disease classification, particularly viral hepatitis markers and autoantibodies such as ANA and AMA. However, free-text examination reports and other clinical narratives may contain contextual information that is difficult to represent through predefined structured variables alone. By encoding these heterogeneous inputs into high-dimensional representations, the LLM-integrated framework provided a way to incorporate narrative information into downstream ML models without requiring extensive rule-based text structuring.

The analysis also supports the value of evaluating models with metrics appropriate for imbalanced multiclass data. Because CHB represented the largest disease group, overall accuracy alone could overemphasize performance in the majority class. We therefore reported macroaveraged precision, recall, and F₁-score, together with class-specific results. These analyses showed that the LLM-integrated models improved macrolevel performance and performed better in non-CHB categories than the ML-only model. This pattern suggests that LLM-derived embeddings contributed most to disease categories with greater clinical heterogeneity rather than simply improving recognition of the dominant CHB group.

Feature importance analyses provided additional support for the complementary role of LLM-derived features. Conventional markers, including viral hepatitis markers, autoantibodies, and liver function indicators, remained among the most informative features. Several LLM-derived principal components were also ranked among the top features, suggesting that the embedding space retained discriminative information not fully captured by structured variables. However, these principal components should be interpreted as compressed semantic representations rather than as directly interpretable clinical concepts.

We also evaluated zero-shot LLM reasoning for liver disease classification and medication recommendation. Although this analysis showed exploratory value, direct LLM reasoning was less stable and less accurate than the embedding-based ML framework, and results should be interpreted with caution given the known sensitivity of zero-shot performance to prompt phrasing and instruction-tuning differences across models. Medication recommendation overlap should also be interpreted cautiously because prescriptions depend on disease severity, contraindications, patient history, and physician judgment. These findings support the use of LLMs as feature encoders within a controlled modeling pipeline rather than stand-alone diagnostic or therapeutic decision systems.

Previous liver disease prediction studies have often relied on selected biochemical markers, serological indicators, or quantitative imaging features [56,57]. These approaches remain clinically meaningful but may not fully capture information contained in free-text reports and other semistructured clinical records. Recent studies have begun to explore LLMs for clinical text understanding, information extraction, and diagnostic reasoning, but their role as embedding-based feature encoders for liver disease classification remains insufficiently studied [58,59].

Our study extends this work by evaluating LLM-derived embeddings in a real-world liver disease cohort and by comparing several modeling strategies, including ML-only models, LLM embedding-only models, LLM-integrated ML models, lightweight natural language processing baselines, and direct zero-shot LLM reasoning. This comparative design helps distinguish the contribution of semantic representation learning from that of downstream ML classifiers. The results suggest that LLM-derived embeddings can improve internal classification performance when integrated with structured clinical variables, especially in diagnostically challenging categories.

This study has several limitations. First, this was a single-center retrospective study, and primary evaluation used an internal holdout validation set rather than an external cohort. Although historical cases were retrospectively readjudicated using updated diagnostic criteria, multicenter external validation is still needed to assess generalizability across hospitals, laboratory systems, and documentation styles. A temporal internal validation using cases from 2010 to 2019 for training and cases from 2020 to 2025 for testing showed that the relative advantage of the LLM-integrated models over the ML-only baseline was preserved (Multimedia Appendix 12). However, this analysis remains an internal validation and does not substitute for external validation across independent centers. Second, predefined exclusions were applied to reduce etiological ambiguity, including the exclusion of patients with multiple coexisting liver disease etiologies and diabetes mellitus. Therefore, whether the framework generalizes to patients with overlapping liver disease mechanisms, substantial metabolic comorbidities, or metabolic dysfunction–associated steatotic liver disease as an independent diagnostic category requires further validation. Third, although missingness thresholds and sensitivity analyses were applied, residual bias from incomplete retrospective EMR data and imputation cannot be fully excluded. In particular, missingness in serological markers such as hepatitis B surface antigen, ANA, and AMA reflects selective test ordering rather than random unavailability. The missing-indicator code used for these variables may therefore partly capture this ordering pattern rather than a purely biological signal, and disentangling the 2 would require prospective or external validation. Fourth, although explicit diagnostic labels, prescribed medication names, and disease-specific treatment keywords were removed or masked before LLM encoding, narrative mentions of specific treatments embedded in free-text reports may not have been completely captured by the rule-based filtering process, representing a potential source of residual label leakage. Fifth, due to hardware constraints, only 8B-parameter LLMs were evaluated, and the impact of larger models or domain-specific fine-tuning on representation quality remains unknown. Sixth, PCA-compressed embedding features cannot be directly mapped to individual clinical concepts, limiting interpretability. Finally, the zero-shot reasoning and medication recommendation analyses were exploratory. Zero-shot performance is sensitive to prompt phrasing, and the use of a single fixed template may have favored certain models. Medication matching was based on prescription overlap and should not be interpreted as evidence of treatment appropriateness or clinical usefulness. Prospective workflow-based evaluation is needed before clinical implementation.

In this single-center retrospective cohort, LLM-derived embeddings provided complementary representations to structured laboratory data for multiclass liver disease classification. The LLM-integrated framework showed improved internal validation performance compared with ML-only models, particularly for non-CHB categories and fine-grained subtype discrimination involving AIH and PBC. These findings support further evaluation of LLM-derived embeddings as part of structured clinical prediction pipelines, but multicenter external validation and prospective assessment are required before integration into clinical workflows.

Future work should evaluate this framework in multicenter cohorts with different laboratory systems, documentation styles, and disease distributions. Prospective studies are also needed to determine whether the model can improve diagnostic workflow, clinician agreement, or subsequent test ordering in real-world settings. Additional work should focus on improving the interpretability of LLM-derived embeddings and assessing whether similar approaches are useful in other clinical domains where free-text reports provide information complementary to structured laboratory data.