The LLM-TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) checklist is provided as Checklist 1. Our fact-checking framework consists of five steps, as shown in Figure 1: (1) generate an initial RAG-based response to the question; (2) split the response into atomic facts; (3) determine the veracity of each fact (categories: “TRUE” and “FALSE”) based on newly retrieved chunks; (4) rewrite the facts detected as incorrect and loop through steps 3 and 4 until all facts are correct, or for a maximum of 3 iterations; (5) rewrite the full response by incorporating the rewritten facts.

All 5 steps are implemented using dedicated LLM prompts, provided in Table S1 of Multimedia Appendix 1. For steps 1 to 4, in-context learning with 4 expert-annotated examples per prompt is used to enforce consistent atomic fact extraction and verdicting. All prompts and examples were created by a medical expert (JCP, 9 y of experience in radiation oncology). Prompts were refined until the performance was deemed satisfactory on a validation set; this included increasing the number of few-shot examples from 0 to 1 and finally to 4. The number of retrieved chunks was set to 7 because in the initial testing it provided the best balance between sufficient useful information and minimizing the noise from irrelevant chunks. An analysis using different numbers of top k chunks (3, 5, 7, and 10) is shown in Table S12 of Multimedia Appendix 1.

The knowledge base consisted of curated oncological guideline documents for prostate and breast cancer (see Table S5 in Multimedia Appendix 1). All PDF documents were converted into plain text using the open-source library PaperMage, which works well with visually rich scientific documents. The text was chunked into overlapping segments of 512 tokens (100-token overlap) and embedded using S-PubMedBERT [13], a transformer model pretrained on PubMed abstracts, giving it an increased semantic understanding of medical concepts, making it highly suitable for our medical use case. Chunks were stored in a ChromaDB vector database. For evidence retrieval, cosine similarity was used to select the 7 most relevant chunks for the question (in step 1) and then the 7 most similar ones for each atomic fact (step 3). All LLM generations were performed using GPT-4o (gpt-4o-2024-11-20; OpenAI) via the OpenAI API with a temperature set to 0 to ensure more deterministic outputs and reproducibility.

While the knowledge base serves as the primary source for both answer creation and verification, the input question itself is also incorporated as a reference source during the fact-checking process. This allows for the verification of patient-specific information included in the query that is not represented in guideline documents. Consequently, patient-specific facts are not incorrectly classified as FALSE only because they cannot be validated against the external knowledge base.

Two additional components of the pipeline were tested. Looping, where the results of one full fact-checking run (with atomic fact corrections) were used as the input to another full run, was tested to determine if it helps correct the facts that were still incorrect. Ensembling, where the fact-veracity prediction (“TRUE” or “FALSE”) was based not on just one prediction output but on multiple predictions using the same LLM and prompt, was used to determine whether fact-veracity prediction could be improved.

The main task of this study was long-form medical question-answering. Two Q&A datasets were constructed by a medical expert (JCP), with each dataset comprising distinct questions about the diagnosis and treatment of prostate or breast cancer. Questions were based on the guideline content provided by the RAG system. They were categorized as fact-based (direct guideline knowledge) or patient-based (clinical vignettes), with varying complexity. To select the optimal prompting strategy, the first set, a validation set of 50 Q&A pairs on prostate cancer, was used. The final configuration was tested on the second set, a separate test set of 60 Q&A pairs (30 prostate and 30 breast cancer).

Asking questions on specialties other than prostate cancer, as in the validation set (implemented breast cancer Q&A in the test set and an independent, established neurology-Q&A set [2] of 65 Q&A items), ensured that no overfitting occurred.

To assess the framework in a realistic clinical setting, 40 anonymized real patient cases from a multidisciplinary tumor board were included. A total of 215 Q&As were used, all of which were checked by human expert evaluators.

All Q&A sets are available in the GitHub (Microsoft) repository [14]. An overview of all Q&As used and their respective questions is available in Table S10 of the Multimedia Appendix 1.

Fact-checking performance was evaluated by comparing assigned verdicts (“TRUE” or “FALSE”) against human assessments. Human evaluation involved verifying the correctness of each atomic verdict and categorizing errors as hallucinations, incorrect information, missing information, or lack of context. Each atomic fact correction itself was analyzed for correctness, while undetected incorrect facts were considered false negatives. Final overall answers were compared to initial responses to determine whether fact-checking led to improvement, deterioration, or no change.

Validation and tumor board analyses were conducted by one medically trained scientist (AD), supervised by JCP. Test-set evaluation used four independent blinded physician raters (AD, FB, JN, and MN), with majority voting; disagreements (2 vs 2) were resolved by a blinded fifth rater (JCP). All four raters agreed in 85% of cases, with ties in 4% of cases. The Fleiss κ and Krippendorff α were both 0.16: this score is an artifact of unbalanced labels—since the LLM classifier was highly accurate, the “1” (correct) label was highly prevalent, making the “agreement expected by chance” very high.

The following metrics were computed: true positive (verdict “FALSE” confirmed by a human), true negative (verdict “TRUE” confirmed by a human), false positive (verdict “FALSE” not confirmed by a human), and false negative (verdict “TRUE” not confirmed by a human). Standard confusion matrix metrics for classification tasks (sensitivity, specificity, precision, F₁, and accuracy) were calculated. Annotator guidelines are available in Multimedia Appendix 1.

The framework was further evaluated on the AMEGA (Autonomous Medical Evaluation for Guideline Adherence) benchmark [15], which tests adherence of LLMs to medical guidelines across 20 clinical domains. This benchmark comprises 20 patient cases (6‐8 questions each; 135 in total), with 1337 predefined evaluation criteria that can be used for automatic evaluation of generated answers. In the original study, responses were improved by iterative “question reasking”. Here, we applied our atomic fact-checking pipeline and compared the initial RAG answers with the corrected answers using the same criteria. The auto-evaluation was done using GPT-4o (gpt-4o-2024-11-20), and for each case, the relevant medical guidelines served as the retrieval database.

Our experiments focused on differences in the fact-checking performance between medical fine-tuned, open-source LLMs and their counterpart general-purpose (ie, nonmedical) models. The compared models include Gemma 3 27B (Google Deepmind) vs MedGemma 27B (Google), Llama 3 70B (Meta AI) vs OpenBioLLM 70B (Saama AI Labs), and Qwen 3 32B (Alibaba Cloud) vs Qwen 3 Medical 32B (Alibaba Cloud). The validation set (50 prostate Q&A pairs) was used, and a human evaluation was conducted as in previous experiments.

In addition to the quantitative fact-checking evaluation, we performed a qualitative automated evaluation. Using the LLM-as-a-judge technique, we defined 7 rubrics that evaluated different aspects of the generated answers’ quality. We used GPT-4.1 as an independent judge model, which produced a numeric score (range 0‐1) for each answer. Each rubric used finely crafted evaluation criteria and steps, which were based on common metrics from a related LLM-as-a-judge framework, G-Eval [16] and adjusted by a medical expert (JCP) where needed. The rubrics used were correctness, completeness, clarity, context faithfulness, coherence, medical harmfulness, and calibration. Evaluation steps are listed in Table S8 of Multimedia Appendix 1.

In order to evaluate how well our atomic fact-checking framework performs compared to other methods, we compared it with the popular framework self-refine [17]. In this framework, the main idea is to first generate an initial response using an LLM, then use the same LLM to provide feedback on how to improve the response, and finally use the same LLM to refine the response based on that feedback. We used the prompts from the original paper, slightly adapted for the Q&A use case (prompts can be found in Table S4 of Multimedia Appendix 1). We evaluated the approach on the AMEGA benchmark using GPT-4o, GPT-4o-mini, Gemma 3 27B, MedGemma 27B, and Llama 3.2 3B.

Institutional review board approval was obtained from the Institutional Review Board of the University Hospital of the Technical University of Munich (ethics approval number 2023‐626_1 S-NP). All patients were treated after obtaining informed consent. Additional informed consent for the scientific study was not necessary due to local legislation (Bayerisches Krankenhausgesetz [18,19]). All patient data were fully anonymized. There was no participant compensation.

Across all evaluation sets, answers were split into a median of 7 (IQR 5‐8.75) atomic facts, yielding a total number of 428, 404, 519, and 474 facts for the validation (50 Q&A), testing (60 Q&A), tumor board (40 Q&A), and neurology (65 Q&A) sets, respectively. Evaluation on the validation set revealed the best architectural strategy as 4-shot prompting for answer generation, verdicting, and fact rewriting (ablation study: Table S7 and Figure S1 in Multimedia Appendix 1). In the atomic fact correction step, retrieving new chunks for each fact, instead of using the initial chunks (used for Q&A) for fact-checking and rewriting, increased the overall performance (both precision and sensitivity).

Looping through all facts labeled as “FALSE” further increased the true positive rate and positive predictive value while reducing falsifications in rewritten facts. The average false positive rate over 3 evaluation sets decreased from 2% to 1% to 0% throughout 3 iterations of the correction loop. Therefore, 3 iterations were chosen as the maximum, since this was enough in experiments to correct all unsupported facts. Ensembling for atomic veracity prediction and changes in temperature during ensembling did not yield a significant gain in performance (Figures S2 and S3 in Multimedia Appendix 1). While it slightly increased the overall F₁-score, it also reduced sensitivity, the most important metric in our system. Therefore, our final pipeline uses 4-shot examples and 3-step looping but no veracity prediction ensembling.

Adding few-shot examples to LLM prompts decreased the precision scores (95, 71, and 61 for 0, 1, and 4-shot settings) but increased the sensitivity (recall) scores (44, 62, and 78). Sensitivity was preferred because it is more important in our system to detect any hallucinations, while any false positives could be dismissed during the rewriting phase.

In the validation-Q&A and test-Q&A sets, this final framework achieved balanced accuracy scores of 87% and 74%, with hallucination detection of 50% and 38% and inaccuracy detection of 58% and 50%, respectively (Table 1). In the more complex tumor board test set, 25% of hallucinations and 24% of inaccuracies were found, with a balanced accuracy of 72%. The positive predictive value for false fact identification was 66%, 100%, and 85% for the validation, test, and tumor board sets, respectively.

Overall answer improvements were seen in 20%, 10%, and 50% of cases in the validation, test, and tumor board–Q&A sets, respectively (Figure 2). Overall answer quality decreased in 8%, 0%, and 7.5 % of cases. The quality of the remaining answers remained the same.

Achieving results comparable to our own test set on the neurology Q&A shows that no overfitting occurred. Presenting with a sensitivity of 52%, a precision of 86%, and a balanced accuracy of 76%, the overall answer quality was improved, remained stable, or worsened in 7.7%, 7.7%, and 3% of cases, respectively (Figure 2).

The AMEGA auto-evaluation analysis revealed a significant improvement in answer quality by applying our fact-checking algorithm in contrast to RAG only for all LLMs tested (P<.001 for 13/16 models, P<.01 for 2/16 models, and P<.05 for 1/16 models; Figure 3; see Table S6 in Multimedia Appendix 1). The overall best answer quality (27.3) among nonreasoning models was achieved by GPT-4o-mini and Mistral 24B, after fact-checking was applied. The largest improvement with fact-checking was seen for Llama 3.2 3B. A significant negative correlation was found between the logarithm of the LLM model parameter size (in billions) and model improvement (Pearson correlation −0.754, P=.03; Figure 4). Even for the reasoning model OpenAI-o1, performance was significantly increased (P<.001), achieving the overall best model performance of 31.52.

An important aspect of the fact-checking framework is the explainability achieved by tracing each atomic fact to the most relevant chunk (passage) in medical guidelines. To assess this, we used the test-Q&A set and compared three similarity definitions of the best-fitting chunks selected from the vector database. A simple chain-of-thought prompt using GPT-4o identified the correct chunk in 75% of cases as the first choice and in 91.9% of cases among the top 3 chunks, outperforming a complex chain-of-thought prompt (instructing the LLM to assign scores and then rank) and cosine similarity with a text-embedding model (see Figure S4 in Multimedia Appendix 1 and Table S3 in Multimedia Appendix 1 for prompts).

In the comparison of open-source medical and nonmedical models (Table 2), MedGemma 27B performed best overall, with a balanced accuracy of 90%, a sensitivity of 83%, and a total improvement in answers of 30%, outperforming even the GPT-4o baseline. The high sensitivity score of MedGemma 27B was better than that of its Gemma 3 27B counterpart and GPT-4o (83% vs 46% vs 78%).

The 2 tested Qwen models showed comparable sensitivity (55% vs 56%), while Llama 3 70B considerably outperformed OpenBioLLM 70B (61% vs 33%). The same tendency holds true for their F₁-scores and balanced accuracies. The positive predictive value was the highest for OpenBioLLM 70B; however, it had markedly reduced sensitivity and an elevated false-negative rate.

Looking at the improved overall answer quality as the main fact-checking outcome, MedGemma 27B demonstrated superior performance compared to its generalist counterpart, Gemma 3 27B, as well as the GPT-4o baseline (30% vs 20% vs 20%) and all other evaluated models. The Qwen models showed moderate results (generalist: 18% and medical: 24%), while Llama 3 surpassed OpenBioLLM (8% vs 4%). Among nonmedical models, Llama 3 achieved the highest balanced accuracy (79%).

Results of LLM-as-a-judge automated rubric evaluation are shown for the baseline, GPT-4o, and the best-performing open-source model based on F₁-score, MedGemma 27B, in Table 3. Results of all open-source models are available in Table S9 of Multimedia Appendix 1. Even though MedGemma 27B outperformed GPT-4o on the fact-checking task (atomic claim veracity prediction), GPT-4o scored better in four rubrics, MedGemma in one rubric, while scores were almost equal in two rubrics. While MedGemma improved more answers in total (30% vs 20%), certain aspects of its answers are still lacking compared to GPT. Most of MedGemma’s answers had significantly higher rubric scores after the fact-checking process was performed. When evaluating all LLMs together, context faithfulness was significantly improved (Table S9 in Multimedia Appendix 1).

Comparison with the competing self-correcting framework self-refine [17] on AMEGA is shown in Table 4. The improvement rate using self-refine depended on the model evaluated. The highest rate was seen for GPT-4o, while Gemini models and open-source models showed lower performance improvements. In comparison with self-refine, fact-checking was significantly better for two smaller open-source models, while there was no difference for recent Gemini models. For GPT-4o, self-refine showed significantly better performance.