Predicting 30-Day Postoperative Mortality and American Society of Anesthesiologists Physical Status Using Retrieval-Augmented Large Language Models: Development and Validation Study

Introduction

Background

Nowadays, postoperative mortality remains a significant concern for health care professionals. Accurate preoperative risk prediction enables better surgical planning and anesthesia management, which can ultimately improve patient outcomes [1-5]. Existing risk assessment tools, such as the American College of Surgeons National Surgical Quality Improvement Program [6] and the American Society of Anesthesiologists physical status (ASA-PS) [7], focus primarily on structured data like patient demographics and medical history. However, these models often exclude valuable information contained in unstructured clinical text, such as surgical notes and physician reports [8].

Prior Work

The use of machine learning (ML) in predicting postoperative mortality has shown promising results. ML models leveraging electronic health records (EHRs) have demonstrated superior predictive performance compared to traditional methods, often achieving a higher area under the receiver operating characteristic curve (AUROC) values [9-13]. Recent advancements in natural language processing, particularly the development of models like Bidirectional Encoder Representations from Transformers (BERT) [14], have improved the ability to extract meaningful insights from unstructured clinical texts [15]. However, BERT’s token length limitation constrains its ability to fully use long-form clinical notes, potentially limiting the model’s effectiveness in real-world applications [14,16].

Recent research has explored the application of large language models (LLMs) in perioperative risk prediction, specifically for predicting ASA-PS classifications. For instance, one study demonstrated that while LLMs, such as GPT-4, achieved moderate F₁-scores both in hospital mortality (F₁-score of 0.86) and ASA-PS predictions (F₁-score of 0.50), their ability to provide interpretability and explanations for the classifications offered potential clinical utility [17]. Despite relatively lower performance in numeric scoring, the use of chain-of-thought reasoning in LLMs allowed for a better understanding of adjacent classifications, assisting clinicians in decision-making during the surgical planning phase [17].

Among LLMs, the LLaMA (LLM Meta AI) series has gained attention for its high performance, open-source availability, and adaptability across diverse natural language processing tasks [18]. LLaMA 3.1, in particular, has been shown to excel in clinical prediction tasks. For example, DRG-LLaMA, a model fine-tuned from LLaMA, demonstrated superior predictive performance for assigning diagnosis-related groups using unstructured clinical notes from the MIMIC-IV dataset. The model achieved an area under the curve of 0.986 and outperformed existing methods like ClinicalBERT, showcasing its utility in handling complex health care–related classification tasks [19].

Furthermore, a systematic review by Pressman et al [20] highlighted the broader potential of LLMs in clinical applications, including diagnostic support and risk stratification through the analysis of unstructured EHR data, underscoring the value of LLMs in perioperative care settings. These findings suggest that LLMs can complement existing risk assessment models by providing valuable insights from unstructured clinical data, despite certain limitations in numeric accuracy.

Aim

This study aimed to demonstrate and evaluate the effectiveness of using an LLM integrated with retrieval-augmented generation (LLM-RAG) to improve clinical risk prediction, specifically for 30-day postoperative mortality and ASA-PS classification, based on preoperative unstructured clinical notes. To this end, we incorporated RAG [21], a technique designed for LLMs that dynamically retrieves and integrates relevant external information during inference. In addition to predicting postoperative mortality, we include the task of ASA-PS classification to evaluate the model’s capabilities in handling preoperative risk stratification [7,22,23]. RAG allows the model to retrieve and incorporate relevant information from external sources during inference, thereby enhancing predictive power [24].

Furthermore, this study explores both zero-shot and few-shot prediction (also known as in-context learning) strategies to investigate their respective impacts on model performance. Few-shot prompting involves providing representative tasks and solution examples in the prompt to guide the model, while zero-shot prompting provides only the query task without examples [25]. By leveraging these approaches, we aim to address the limitations of current models—particularly in handling long clinical notes, improving interpretability [24,26], and enhancing generalization across multiple tasks. Through this research, we seek to demonstrate the potential for improved prediction accuracy, model interpretability [27], and better clinical decision-making in real-world hospital settings [28]. We expect that, by improving risk prediction and enabling prompt‑driven interpretability, this LLM‑RAG approach could enhance perioperative decision support.

Methods

Ethical Considerations

This retrospective observational study was approved by the institutional review board of Far Eastern Memorial Hospital (112166-F), a large academic medical center. Preoperative anesthesia assessment records, surgical information, and discharge summaries were collected from the hospital’s EHR from January 1, 2016, to July 31, 2023.

Inclusion and Exclusion Criteria

Patients included in this study were aged 18 years or older and had undergone at least one surgical procedure under general or neuraxial anesthesia (n=118,274). Cases were excluded if they had an ASA-PS classification of 6 (n=90), indicating patients declared brain-dead for organ donation. Additionally, records were excluded if they lacked critical information, including entry time (n=2), exit time, preoperative diagnosis, proposed procedure text (n=3), or those whose surgery date fell after all discharge dates (n=93,688). The final cohort consisted of 24,491 patients and was divided into training (n=19,592) and validation cohorts (n=4899; Figure 1).

‎

Outcome Definition

The primary endpoint of this study was in-hospital 30-day postoperative mortality, defined as any patient death occurring within 30 days after surgery. Mortality status was determined based on discharge records explicitly labeled as “expired” or “critical against-advice discharge” (ie, patients who left the hospital against medical advice in a critical condition, with no subsequent readmission recorded within the 30-day window). Patients discharged alive and not readmitted within 30 days were classified as nonmortality cases.

The secondary endpoint was the ASA-PS classification, recorded at the time of preoperative anesthesia evaluation. ASA-PS was categorized into five classes (labeled as 1-5), reflecting the patient’s physical condition prior to surgery [7,22,23]:

• Class 1 (ASA I): A normal, healthy patient with no systemic disease. This category includes individuals who are free from any significant health conditions and are at the lowest risk for anesthesia-related complications.
• Class 2 (ASA II): A patient with mild systemic disease that does not limit physical activity. Examples include controlled hypertension or diabetes without systemic complications, mild obesity, or smoking without chronic obstructive pulmonary disease.
• Class 3 (ASA III): A patient with severe systemic disease that limits physical activity but is not incapacitating. Conditions may include poorly controlled hypertension, diabetes with systemic complications, or a history of myocardial infarction.
• Class 4 (ASA IV): A patient with severe systemic disease that is a constant threat to life. Examples include recent myocardial infarction, unstable angina, advanced liver or kidney disease, or severe heart failure.
• Class 5 (ASA V): A moribund patient who is not expected to survive without the operation. These patients typically have life-threatening conditions such as a ruptured abdominal aneurysm or massive trauma.

Data Preprocessing

Outlier Filtering

Continuous variables such as height, weight, and BMI were filtered to remove physiologically implausible values (eg, height: 120-250 cm; weight: 25-200 kg; BMI ≤100 kg/m²). Laboratory data, including hemoglobin, hematocrit, blood sugar, and potassium, were filtered for outliers. Vital signs, including heart rate, blood pressure, respiratory rate, body temperature, and pulse oximetry, were filtered for outliers (Multimedia Appendix 1). Outliers were identified as potential input errors and treated as missing data.

Structured Data Preprocessing

Structured features included patient characteristics, surgical characteristics, comorbidities, preoperative laboratory values, and preoperative vital signs (Table 1). Continuous features included age, height, weight, latest laboratory data before surgery (eg, hemoglobin, platelet count, and creatinine), and preoperative vital signs (eg, body temperature, oxygen saturation, heart rate, respiratory rate, and systolic and diastolic blood pressure). Missing values were imputed with the median value of the dataset for continuous features.

Categorical features with only 2 classes, such as sex, comorbidities (eg, diabetes mellitus and hypertension), ASA emergency status, and consciousness status, were converted into binary encoding. Features with more than 2 classes, such as ASA-PS (5 classes), department (22 classes), emergency level (4 classes), preoperative location (4 classes), and anesthesia type (4 classes), were transformed into one-hot encodings. For all categorical features, missing values were imputed using the majority category from the training dataset.

Unstructured Data Preprocessing

Unstructured data were primarily used in the LLaMA-RAG model, leveraging LLMs, using RAG or not, and using few-shot or one-shot. These free-text clinical notes included provider service, procedure, diagnosis, emergency level, planned anesthesia, description, chief complaint, present illness, discharge diagnosis, and discharge treatment. Provider service (department), emergency level, and planned anesthesia (anesthesia type) were originally recorded in text but were also used in categorical features above after preprocessing. The integration of unstructured data allowed us to evaluate their ability to provide additional clinical insights and improve predictive accuracy beyond what structured data could achieve.

For each surgery, we extracted the latest preoperative anesthesia assessment and the discharge summary preceding the operation, along with up to four earlier summaries to capture longitudinal context. All text underwent cleaning to remove templates and headers, was converted to lowercase, stripped of punctuation, and uniformly tokenized. To meet the 4096-token limit of LLaMA 3.1‑8B, we first used a summarization prompt (Multimedia Appendix 2) to summarize each discharge note into its key clinical elements, then concatenated these summaries in chronological order. If the combined narrative exceeded the token limit, we truncated from the oldest summaries, preserving the most recent information. This pipeline distilled rich longitudinal context into a compact, structured input optimized for downstream LLM-based prediction tasks.

Model Development

LLM Setup

Figure 2 outlines our LLaMA‑RAG experimental framework. We split the full cohort into an 80% training and a 20% validation set. The lower section illustrates the workflow with RAG and the upper section without. During validation, cases with only 1 prior record were fed directly as zero‑ or few‑shot prompts, whereas those with up to 4 prior records were first summarized using a summarization prompt and then concatenated in chronological order before prompting. For RAG configurations, each case’s query was also sent to a vector database of domain text—chunked, embedded, and retrieved via semantic search—and the top‑ranked passages were appended to the prompt.

‎

To ensure that the selected model aligns with our needs, we conducted preliminary comparisons between LLaMA 3.1-8B and LLaMA 3.2-3B using identical prompts and hyperparameters (eg, temperature, top_p) [18]. We selected these two models due to their ability to operate efficiently on available computational resources, including the V100 graphic processing unit, 4 central processing unit cores, and 90 GB of memory. The results showed that LLaMA 3.1-8B consistently outperformed its smaller counterpart under the same conditions. Specifically, for the ASA classification task, the micro F₁-score improved significantly from 0.3246 (95% CI 0.3222-0.3246) with LLaMA 3.2 to 0.4544 (95% CI 0.4531-0.4548) with LLaMA 3.1. Similarly, for the mortality prediction task, the weighted area under the precision-recall curve (AUPRC) exhibited an increase from 0.4311 (95% CI 0.4253-0.4369) with LLaMA 3.2 to 0.4959 (95% CI 0.4828-0.5090) with LLaMA 3.1 [29,30]. Other performance metrics, including precision, recall, and AUROC, also demonstrated consistent improvements and are detailed in Multimedia Appendix 3. Based on these findings, LLaMA 3.1-8B was chosen as the primary model for this study.

Prompting Strategies

We used four prompting strategies to evaluate the model’s performance:

• No RAG, no few-shot: Zero-shot Q&A using original clinical notes without any prior examples or retrieval.
• No RAG, with few-shot: Incorporating one or more prior examples in the prompt but without RAG.
• RAG, no few-shot: Zero-shot Q&A with RAG using external knowledge retrieved from Miller’s Anesthesia, 2-Volume Set, 9th Edition (2020) [31].
• RAG, with few-shot: Combining RAG with few-shot prompting, where external knowledge retrieved from Miller’s Anesthesia, 2-Volume Set, 9th Edition (2020) [31] is used to augment the prompt.

Prompts included structured tags for role assignment, task specification, retrieved information summaries, and output formatting. Full prompt examples are shown in Figure 3.

‎

Experimental Setup

We incorporated experiments using ML models—including random forest [32], eXtreme Gradient Boosting (XGBoost) [33], logistic regression [34], and support vector machines (SVM) [35]—which used structured data as input features, in contrast to the unstructured clinical notes used by the LLM-based RAG framework. To ensure a fair and robust comparison, we systematically optimized the hyperparameters of these ML models using grid search and randomized search, implemented via the Scikit-learn library [36]. Grid Search exhaustively evaluates all possible hyperparameter combinations within a predefined space, ensuring optimal model configurations, whereas randomized search provides a computationally efficient alternative by sampling hyperparameter values from a given distribution [37]. Hyperparameter tuning was conducted using 5-fold cross-validation, optimizing for weighted AUPRC in the mortality prediction task and micro F₁-score in the ASA classification task, ensuring that each model was evaluated under its best-performing configuration.

Experiments compared the LLaMA-RAG model configurations against ML baselines. Performance was benchmarked using the same training and validation cohorts, consistent preprocessing, and matched evaluation criteria. All experiments were executed on a V100 graphic processing unit with 4 central processing unit cores and 90 GB of memory.

Model Evaluation and Statistical Analysis

This study used a retrospective cohort design with a total sample size of 24,491 surgical cases, split into a training cohort (n=19,592) and a validation cohort (n=4899). No a priori sample size or power calculation was performed; instead, the study included all eligible cases within the data collection period (January 2016-July 2023).

Model performance was assessed using the F₁-score for 30-day mortality prediction and micro F₁-score for ASA classification. Additional metrics, including accuracy, precision, recall, and specificity, were reported. Statistical significance of model comparisons was assessed post hoc using bootstrapping with 1000 iterations, defining significance as a 95% CI for the performance difference excluding zero (P<.05).

We applied Shapley Additive Explanations (SHAP) to our structured-data ML models to quantify and visualize each feature’s contribution to both 30-day postoperative mortality and ASA-PS classification, generating global importance bar plots and summary dot plots for interpretability [38].

Results

Outcome Distribution

In the final dataset of 24,491 unique patient records, a total of 520 (2.1%) patients experienced in-hospital 30-day postoperative mortality, including 424 (2.2%) deaths in the training (n=19,592) and 96 (2%) deaths in the validation cohort (n=4899). The ASA-PS class distributions were as follows: in the overall cohort (n=24,491), Class 1 (n=535, 2.2%), Class 2 (n=15,272, 62.4%), Class 3 (n=8024, 32.8%), Class 4 (n=606, 2.5%), and Class 5 (n=54, 0.22%); in the training cohort (n=19,592), Class 1 (n=433; 2.2%), Class 2 (n=12,207, 62%), Class 3 (n=6428, 33%), Class 4 (n=481, 2.4%), and Class 5 (n=43, 0.22%); in the validation cohort (n=4899), Class 1 (n=102, 2.1%), Class 2 (n=3065, 63%), Class 3 (n=1596, 33%), Class 4 (n=125, 2.6%), and Class 5 (n=11, 0.22%). These distributions were balanced across training and validation cohorts to support reliable model development and evaluation (Table 2 and Multimedia Appendix 4).

Preliminary Hyperparameter Tuning for the LLaMA + RAG Model

Before evaluating the performance of our proposed LLaMA + RAG model against ML approaches, we conducted a series of preliminary experiments to determine the optimal configuration for embedding models, retrieval settings, and hyperparameters. Specifically, we explored:

• Embedding models: MedEmbed [39] and PubMedBERT [40]
• Chunk sizes: 150, 200, 250, 300, and 350
• Retrieval top-k values: 6, 8, and 12
• Temperature and top-p settings: (0.001, 0.001), (0.1, 0.1), and (1, 0.5)

The goal of these experiments was to systematically identify the most effective parameter combinations for mortality prediction and ASA classification, ensuring a fair comparison between LLaMA + RAG and traditional ML models.

Our findings revealed that:

• For mortality prediction, the best-performing configuration was MedEmbed, chunk size=250, temperature=1, top_p=0.5, retrieval top_k=8.
• For ASA classification, the optimal setup was MedEmbed, chunk size=250, temperature=0.001, top_p=0.001, retrieval top_k=8.

The complete results of these parameter selection experiments, including comparisons across different settings, are provided in Multimedia Appendix 5.

With these optimal settings, we proceeded to evaluate our LLaMA + RAG model against ML algorithms, including random forest, XGBoost, SVM, and logistic regression. The following sections present a comparative analysis of mortality prediction and ASA-PS classification, demonstrating the advantages of integrating RAG with LLMs for clinical risk prediction.

Performance Comparison

To evaluate the effectiveness of our proposed LLaMA-based method using unstructured clinical notes, we compared its performance against ML models trained on structured data. The comparison focused on mortality prediction and ASA-PS classification, using evaluation metrics including F₁-score, accuracy, precision, recall, and specificity. The detailed results are presented in Table 3 for mortality prediction and Table 4 for ASA-PS classification. Additionally, a more comprehensive set of performance metrics is provided in Tables S1 and S2 in Multimedia Appendix 3 to offer deeper insights into model performance across various evaluation perspectives.

Mortality Prediction Task

Our LLaMA-RAG model achieved the highest F₁-score of 0.4663 (95% CI 0.4654-0.4672) and recall of 0.9375 (95% CI 0.9341-0.9409), indicating its strong capability in correctly identifying mortality cases (Table 3). Among ML models, XGBoost with randomized search attained the second-highest F₁-score of 0.0.4459 (95% CI 0.4176-0.4742), while SVM with randomized search obtained the second-highest recall of 0.8542 (95% CI 0.8478-0.8606). In ablation experiments, the F₁-score dropped to 0.2369 (95% CI 0.2341-0.2397) without few‑shot prompting, 0.0879 (95% CI 0.0717-0.1041) without RAG, and 0.0436 (95% CI 0.0292-0.0580) without either few-shot prompting or RAG.

Attention heatmaps reveal that, for mortality prediction, the LLaMA-RAG model consistently concentrates its attention on the “present illness” section, often supplemented by the “discharge diagnosis” field, underscoring acute clinical trajectory as the primary driver of mortality risk estimation (Multimedia Appendices 6 and 7). SHAP analysis of the XGBoost revealed that hemoglobin was the single most influential predictor—lower values markedly increasing predicted risk—followed by intensive care unit, tachycardia, and serum sodium, all of which exhibited the highest mean absolute SHAP values among the top 20 features (Multimedia Appendix 8).

ASA Classification Task

For ASA classification, our LLaMA-RAG model outperformed ML models, achieving the highest micro F₁-score of 0.8409 (95% CI 0.8238-0.8551) and specificity of 0.8836 (95% CI 0.8834-0.8838), detailed in Table 4. Among traditional methods, XGBoost with grid search achieved the second-highest micro F₁-score of 0.8273 (95% CI 0.8209-0.8498), whereas SVM with randomized search obtained the second-highest specificity of 0.8569 (95% CI 0.8566-0.8572). In ablation experiments, the micro F₁-score dropped to 0.6546 (95% CI 0.6430-0.6796) without few‑shot prompting, 0.6340 (95% CI 0.6157-0.6535) without RAG, and 0.4238 (95% CI 0.3952-0.4490) without either few-shot prompting or RAG.

Attention heatmaps for ASA-PS classification show that the LLaMA-RAG model consistently focuses its attention on the “present illness” section, often with additional peaks on “discharge diagnosis” and “discharge treatment,” indicating that the model leverages the patient’s most recent clinical trajectory and outcomes when assigning ASA-PS (Multimedia Appendices 6 and 7). SHAP analysis of the XGBoost revealed that age was the single most influential predictor, lower values markedly decreased predicted class, followed by elective surgery, cardiac disease, and cardiovascular surgery, all of which exhibited the highest mean absolute SHAP values among the top 20 features (Multimedia Appendix 8).

Discussion

Principal Results

This study demonstrates the effectiveness of integrating an LLM with RAG for clinical risk prediction, achieving superior performance in both mortality prediction and ASA classification.

In the mortality prediction task, our LLaMA-RAG model obtained the highest AUPRC of 0.6536 (95% CI 0.6479-0.6593), AUROC of 0.9570 (95% CI 0.9543-0.9597), and F₁-score of 0.4663 (95% CI 0.4654-0.4672), significantly outperforming ML models, including random forest, SVM, XGBoost, and logistic regression in AUPRC (Multimedia Appendix 9). While our AUROC score was significantly higher than that of SVM and logistic regression, it remained comparable to that of random forest and XGBoost (Table 3). For ASA multiclassification, our method achieved the highest micro F₁-score of 0.8409 (95% CI 0.8238-0.8551), outperforming all other ML models (Table 4). In ablation experiments for both tasks, removal of either RAG or few‑shot prompting led to substantial declines in the F₁-score (Tables 3 and 4), underscoring the critical contributions of both components to the model’s discriminative performance.

However, while our approach demonstrated advantages in AUPRC, AUROC, and recall, it did not surpass all ML baselines in terms of accuracy and precision. A closer examination of confusion matrices (Tables S1 and S2 in Multimedia Appendix 3) provides insight into this result: our model exhibited superior sensitivity in identifying rare cases, such as ASA Class 5 patients and postoperative mortality. Even when misclassifications occurred, they were predominantly within adjacent ASA classes, which may reflect the inherent subjectivity in ASA grading (Multimedia Appendix 10). This suggests that our method has the capability to perform clinical risk stratification.

Comparison With Prior Works

Our method also outperformed the approach of Chung et al [17], which used ChatGPT-4 Turbo for ASA classification, achieving a micro F₁-score of 0.8409 (95% CI 0.8238-0.8551), compared to their best result of 0.54 (95% CI 0.49-0.60). This difference highlights the limitations of a general-domain LLM without structured retrieval augmentation. While GPT-4 Turbo benefits from extensive pretraining on diverse text sources, its lack of domain-specific retrieval may restrict its ability to accurately capture complex clinical relationships. By contrast, our retrieval-enhanced model systematically integrates domain knowledge, leading to a more interpretable and clinically reliable risk stratification process [41,42].

When compared to prior deep learning–based approaches, our method significantly outperforms the BERT–deep neural network (DNN) model of Chen et al [43] in mortality prediction. Specifically, our LLM with RAG integration achieved a substantially higher macro F₁-score (0.7222, 95% CI 0.6998-0.7446) compared to their model (0.307, 95% CI 0.269-0.342), indicating a superior balance between precision and recall across both mortality and survival classes. Unlike the BERT-DNN framework, which relies solely on parametric memory for text extraction, our approach dynamically retrieves specialized anesthesia knowledge from textbooks [31]. This retrieval mechanism not only enhances contextual understanding but also mitigates the risk of hallucinated responses common in LLMs [41,42]. By leveraging domain-specific knowledge, our method achieves more reliable and clinically relevant predictions.

In addition to performance comparisons, it is important to note that our study leveraged a substantially larger and more diverse dataset (n=24,491) compared to Chung et al [17], whose task-specific datasets were limited to 1000 cases each. This broader dataset allowed us to capture a wider range of surgical procedures, patient backgrounds, and rare outcomes, enhancing the generalizability of our model. Furthermore, unlike Chung et al [17], who focused primarily on few-shot prompting strategies without retrieval augmentation, our integration of RAG provided domain-specific knowledge, significantly improving predictive accuracy and interpretability. Although Chen et al [43] trained a BERT-DNN model on 121,313 cases, their approach relied on structured inputs and brief diagnostic and procedural snippets (less than 50 tokens) without RAG. By contrast, our framework leveraged both structured features and rich, LLM-summarized discharge narratives to evaluate LLaMA-RAG for 30-day mortality and ASA-PS prediction.

Key Factors Influencing Model Performance

Our findings highlight the critical role of RAG in improving LLM-driven clinical predictions. Traditional LLMs rely solely on their pretraining corpus, which limits their ability to provide up-to-date and domain-specific medical insights. By dynamically retrieving relevant knowledge from trusted medical references, RAG ensures that predictions remain factually grounded and clinically relevant [41,42]. This advantage was particularly evident in mortality prediction, where the incorporation of RAG helped mitigate class imbalance issues, leading to improved AUPRC and predictive performance (Table 3).

To further investigate the effectiveness of RAG-enhanced LLMs, we systematically evaluated 3 key factors influencing model performance: embedding model selection, chunk size configuration, and few-shot prompting strategies (Table S3 in Multimedia Appendix 3 and Multimedia Appendix 11).

Impact of Embedding Model Selection

We first compared MedEmbed and PubMedBERT under identical hyperparameter settings to assess their impact on model performance. Our results demonstrated that MedEmbed consistently outperformed PubMedBERT, achieving the highest AUPRC of 0.4733 for mortality prediction and the highest micro F₁-score of 0.6307 for ASA classification (Figures S1 and S2 in Multimedia Appendix 5). These findings suggest that embedding models optimized for medical text representations can significantly enhance LLM-based clinical predictions.

Effect of Chunk Size on Model Performance

Next, we examined the influence of chunk size selection in retrieval-based augmentation. Our experiments revealed that a chunk size of 250 yielded the best performance across both tasks. For mortality prediction, the highest AUPRC (0.4733) was obtained at retrieval top_k=12, whereas for ASA classification, the best micro F₁-score (0.6307) was achieved at retrieval top_k=6 (Figures S3 and S4 in Multimedia Appendix 5). These results emphasize that optimizing the chunking strategy when embedding clinical texts plays a crucial role in maximizing retrieval effectiveness and overall predictive accuracy.

Influence of Few-Shot Learning on Performance

We further assessed the impact of few-shot prompting, which provided additional performance gains. For mortality prediction, incorporating 9-shot examples resulted in the highest AUPRC of 0.6536, compared to 0.6176 with 5-shot prompting. In ASA classification, few-shot learning significantly improved model performance, yielding a micro F₁-score of 0.8409, a substantial increase over the 0.5938 obtained without few-shot prompting (Figures S5 and S6 in Multimedia Appendix 5). These findings underscore the importance of carefully curating Few-Shot Examples to enhance the generalizability of LLM-based clinical models.

Implications for Clinical Artificial Intelligence

Collectively, our findings highlight the benefits of integrating RAG with LLMs, selecting an effective embedding model, optimizing chunk size, and leveraging few-shot prompting to improve clinical prediction performance. The ability to dynamically retrieve domain-specific knowledge makes RAG-enhanced LLMs particularly well-suited for real-world medical decision-making, where contextual interpretation of patient characteristics and clinical guidelines is essential (Multimedia Appendix 12). This capability is particularly advantageous in medical domains where well-established guidelines and textbooks serve as primary decision-making references. Unlike conventional LLMs that rely solely on pretraining data, RAG dynamically retrieves up-to-date clinical knowledge, ensuring that predictions and recommendations remain aligned with current best practices. By mitigating the risks of outdated information and hallucinated responses, RAG enhances the reliability and interpretability of artificial intelligence (AI)–driven decision support.

Additionally, our confusion matrices suggest that while our method may not always outperform traditional ML models in precision and accuracy, it excels at identifying rare but clinically significant cases, such as ASA Class 5 patients and postoperative mortality events (Multimedia Appendix 13). For ASA class 5, the full LLaMA-RAG model achieved a true positive rate of 54.5%, compared with 54.5% without RAG, 27.3% without few-shot prompting, and 9.1% without either component; traditional ML models identified no class V cases (0% true positives). For 30-day mortality, LLaMA-RAG reached a true positive rate of 94.8%, versus 87.5% without RAG, 79.2% without few-shot prompting, and 56.2% without either, while ML baselines ranged from 35.4% to 85.4%. These results demonstrate that both RAG and few-shot prompting incrementally enhance the LLaMA-RAG model’s ability to detect rare but clinically significant outcomes. This highlights the trade-off between precision and recall in clinical decision support systems, emphasizing the need to balance risk prediction sensitivity and specificity in practice.

Moreover, these results underscore the transformative potential of LLM integrated with RAG architectures in clinical informatics. By improving both discrimination and interpretability, RAG-enhanced LLMs provide a more reliable and adaptable framework for clinical AI applications, bridging the gap between machine intelligence and human expertise in medical practice.

Limitations

This study used a fixed-length chunking strategy for RAG, which may misalign with the semantic structure of a clinical text. Semantic chunking, leveraging meaning-based segmentation, could enhance retrieval relevance [44,45]. Adaptive retrieval, incorporating hybrid query-based methods, may further optimize context selection [45]. Additionally, Adaptive RAG, which adjusts retrieval depth based on query complexity, remains unexplored in our setting but has shown promise in dynamically optimizing information retrieval [46]. Future studies should systematically evaluate these techniques to improve retrieval precision, computational efficiency, and applicability in clinical AI systems.

Conclusions

This study demonstrates the effectiveness of integrating LLMs with RAG for clinical risk prediction. Our LLaMA-RAG model demonstrated superior performance compared to both ML models and previously published deep learning approaches in mortality prediction and ASA classification [17,43], especially for rare high‑risk patients, by grounding its analysis in reliable medical knowledge to deliver more accurate, interpretable, and clinically relevant predictions. Through systematic evaluations of embedding model selection, chunk size optimization, and few-shot prompting, we identified key factors influencing predictive accuracy. RAG-enhanced LLMs provide a promising pathway toward more interpretable, context-aware, and accurate clinical decision support systems.