This study was based exclusively on publicly available data, including published clinical guidelines and peer-reviewed papers. As the research did not involve human participants, it was exempt from Institutional Review Board review, and requirements for informed consent, participant privacy, and compensation were not applicable. All patient information referenced in the source literature was presumed to have been deidentified by the original authors in accordance with established ethical standards.

Our evaluation protocol involved manual annotation by 2 domain experts for the key dimensions of accuracy, honesty, and error composition. To first establish the reliability of these subjective assessments, we calculated the interrater reliability before any performance analysis. We achieved almost perfect agreement for accuracy (Cohen κ=0.903) and honesty (κ=0.954), and substantial agreement for error composition (κ=0.837), confirming the robustness of our evaluation methodology.

Our primary analysis focused on the overall accuracy of the 3 evaluated LLMs across the basic (Figure 2A) and advanced (Figure 2B) tiers. Within each tier, we compared LLM performance under 6 distinct prompting strategies in 2 evaluation modalities: multiple-choice (hatched bars) and open-ended (solid bars).

A primary finding is the significant impact of evaluation modality. This “scaffolding effect," a term from cognitive science, is shown in Figure 2. Across nearly all conditions, accuracy in the open-ended modality was substantially lower than in multiple-choice. This drop was most pronounced for non-RAG methods in the basic tier (Figure 2A), where Claude Opus 4 with the vanilla prompt declined from 93.6% to 71.6%. Predefined selection options transform a demanding generative recall task into a simpler discriminative recognition task; removing this scaffold exposes the LLM’s limitations in open-ended scenarios.

This analysis also highlighted the clear superiority of the RAG strategies, particularly when the “scaffolding” was removed. In the multiple-choice modality on the basic tier (hatched bars in Figure 2A), basic RAG was already a top-performing strategy, achieving accuracies of up to 97.2%. However, the superiority of RAG became most pronounced in the more challenging open-ended modality. Across both tiers (solid bars in Figures 2A and B), all RAG-based methods—basic, advanced, and hybrid RAG—consistently and significantly outperformed their non-RAG counterparts. For example, in the advanced open-ended setting (Figure 2B), Claude Opus 4’s accuracy increased substantially from 63.6% with RoT to 89.3% with advanced RAG and further to 89.7% with hybrid RAG, representing one of the most substantial performance gains observed in this setting. This underscores that for realistic, open-ended tasks, providing precisely aligned external knowledge is a highly effective strategy for improving accuracy.

However, our results revealed that the effectiveness of specialized RAG is highly dependent on the alignment between the retrieved context and the question’s knowledge domain. This dependency was clearly illustrated by the models’ performance when presented with out-of-domain context. In the advanced multiple-choice setting (hatched bars in Figure 2B), a notable adverse effect was observed for the Claude Opus 4 model, whose accuracy declined significantly to 72.8% when using the out-of-domain basic RAG strategy. A converse pattern was observed in the basic multiple-choice setting (hatched bars in Figure 2A), where the introduction of out-of-domain advanced RAG context also led to a performance drop for some models, such as GPT-4o, declining to 63.2%.

The introduction of the hybrid RAG strategy directly addresses this domain alignment challenge. By having access to both knowledge bases, hybrid RAG simulates a more practical scenario. The results demonstrate its remarkable robustness. In the basic setting (Figure 2A), hybrid RAG’s performance nearly matched that of the specialized basic RAG (eg, 95.2% vs 95.6% for GPT-4o in the multiple-choice modality). Similarly, in the advanced setting (Figure 2B), it remained highly competitive with the top-performing advanced RAG. This indicates that the hybrid RAG approach successfully navigates the knowledge selection challenge, maintaining high accuracy without requiring prior knowledge of the question’s domain. It effectively mitigates the performance degradation seen with misaligned RAG, positioning it as a more generalizable and practical solution for real-world applications.

To further dissect the LLM performance, we analyzed its accuracy across different clinical task categories (Figure S1 in Multimedia Appendix 1). The “scaffolding effect” of selection options was immediately apparent. A visual comparison between the multiple-choice modality (Figure S1A in Multimedia Appendix 1) and the open-ended modality (Figure S1B in Multimedia Appendix 1) reveals a universal degradation of performance across nearly all tasks and LLMs. The radar plots in the open-ended setting are visibly “shrunken,” indicating that the performance drop observed in the overall accuracy is a pervasive phenomenon affecting all facets of clinical tasks.

Our task-specific analysis provides a granular capability profile of current LLMs, offering actionable insights for both biomedical researchers and clinicians.

For researchers, LLMs demonstrated considerable reliability on fact-retrieval–oriented tasks, such as those in the “Molecular and Genetic Basis” category (Figure S1A in Multimedia Appendix 1). This suggests their potential as powerful hypothesis generation assistants. For example, a researcher could rapidly query an LLM about the known molecular features of MSI tumors across different cancer types, accelerating literature review and experimental design.

For clinicians, the profile reveals a critical dichotomy. On one hand, LLMs can serve as reliable and rapid knowledge retrieval tools for well-established facts, such as those in “Diagnosis and Testing.” On the other hand, we pinpointed areas where clinicians must exercise extreme caution. A consistent pattern of underperformance was observed in complex, multifactor decision-making tasks, with “Therapeutic Response (Chemotherapy)” emerging as a prominent failure mode across all models in the challenging open-ended setting (Figure S1B in Multimedia Appendix 1). For instance, under the vanilla prompt, the accuracy of GPT-4o on this subtask (n=21) was only 33.3%, a performance level statistically indistinguishable from random chance in a high-stakes context. This striking weakness indicates that when faced with nuanced clinical scenarios that require weighing competing evidence, the internal knowledge of general-purpose LLMs is dangerously unreliable. This is the domain where unassisted LLM usage poses the greatest risk.

Crucially, our analysis also validates a clear pathway to mitigate these risks through a well-designed RAG system. However, the journey to effective RAG is nuanced. The erratic performance of basic RAG, which sourced knowledge only from general guidelines, highlights the danger of grounding models in incomplete or overly broad information; in some advanced tasks, it even underperformed non-RAG methods. In contrast, advanced-RAG, which used our specialized MSICKB knowledge base, consistently improved performance across most high-risk tasks. The most robust results were achieved with hybrid RAG, which integrated both knowledge sources, leading to substantial and stable performance gains, particularly in the most challenging decision-making areas such as chemotherapy response. This improvement was statistically significant for hybrid RAG with both GPT-4o (P<.001; McNemar test) and Gemini 2.5 Pro (P=.02) compared to the vanilla strategy, although not for Claude Opus 4 (P=.22), reflecting its unique interaction with RAG systems.

This finding provides a clear directive: for high-stakes clinical decision support, RAG is not a monolithic solution but a framework whose effectiveness is contingent upon the quality and comprehensiveness of its knowledge base. A well-curated, specialized knowledge source is an essential safety and performance mechanism.

Moving from performance to safety, we analyzed the composition of incorrect responses by categorizing them into 3 distinct behaviors: fabrication, justified refusal, and false refusal (Figure 3). Our analysis begins with the basic tier (Figures 3A and 3B), where a stark pattern emerged: nearly all incorrect responses were classified as fabrication. This outcome, however, is not an indicator of inherent model dishonesty but a direct consequence of the benchmark’s design. The selection-based and true or false questions in this tier did not include a “don’t know” option, structurally forcing the LLM to guess when faced with uncertainty and precluding any form of refusal.

The models’ true behavioral tendencies are revealed in the advanced tier (Figures 3A and 3B), where expressing uncertainty is possible. Even here, non-RAG strategies (vanilla, CoT, and RoT) demonstrate a strong inclination toward fabrication, with rates often exceeding 85%. This highlights a critical safety concern, that is, without a bounded context, LLMs tend to guess rather than admit ignorance. In contrast, the introduction of RAG strategies fundamentally alters this behavior. Across all models and modalities, RAG dramatically reduces the proportion of fabrication by converting these high-risk errors into justified refusals. This shift represents a significant enhancement in model honesty, as the LLMs learn to correctly identify when the provided context is insufficient to form an answer.

However, this safety improvement comes at a price: the emergence of “false refusals,” a failure mode unique to RAG where the model refuses to answer despite the context containing the necessary information. This dynamic is most vividly illustrated by Claude Opus 4 under the basic RAG strategy in the multiple-choice setting (Figure 3A). Here, fabrication plummeted from 97.2% (vanilla) to just 35.2%. This reduction was primarily achieved by converting a remarkable 57.7% of errors into desirable justified refusals. The cost, however, was a 7% rate of false refusals. This pattern—substituting dangerous fabrications with mostly safe justified refusals at the expense of some loss in utility—is a consistent and critical finding of our RAG analysis.

This analysis demonstrates that RAG is a powerful mechanism for inducing epistemic caution. Grounding the model in a bounded context provides a clear basis for assessing knowledge limits, which is the mechanism behind justified refusals. While the existence of false refusals highlights a persistent challenge in ensuring models robustly use all provided evidence, the net effect represents a highly favorable shift in the error profile. RAG systematically replaces high-severity, factually incorrect errors (fabrication) with manageable, lower-severity utility errors (false refusals), thereby making the LLM’s failure modes safer and more trustworthy.

To understand the root causes of errors, we conducted a detailed error composition analysis, applying distinct taxonomies for non-RAG and RAG methods to precisely identify failure points (Figures 4A and 4B).

For methods relying on the LLM’s internal knowledge (vanilla, CoT, and RoT), a clear pattern emerged: failures were overwhelmingly caused by an internal knowledge deficit. This confirms that for this specialized domain, the inherent knowledge gaps of general-purpose LLMs are the primary barrier to accuracy.

The introduction of RAG altered the error distribution. The proportion of internal knowledge deficit errors was substantially reduced across all RAG configurations, as the external knowledge base compensated for the models’ internal deficiencies.

However, this shift revealed that retrieval failure replaced knowledge deficit as the new dominant source of error. In most RAG scenarios, retrieval failure constituted the single largest component of incorrect responses, often accounting for over half of all errors. This indicates that while RAG addresses the problem of “not knowing,” it introduces the challenge of “not finding,” making system performance highly dependent on the accuracy of the retrieval module.

Other RAG-specific errors, such as context ignorance (the LLMs failing to use correct context) and reasoning errors, occurred at a lower frequency. This suggests that when the retriever succeeds in providing the correct context, the LLMs are generally capable of using it appropriately. Therefore, a key area for improving RAG performance in this domain is to enhance the retriever’s ability to consistently deliver the correct context.

Finally, we evaluated the token usage of each strategy by measuring the total token count (input and output) for each interaction (Figures 5 and 6).

The most notable finding, illustrated by the average token counts in Figure 5, was the substantial token usage associated with complex reasoning prompts. The RoT strategy consistently consumed the most tokens across all conditions. This effect was particularly pronounced for Gemini 2.5 Pro in the multiple-choice–Basic setting (Figure 5A), where the average total tokens for RoT (1949 tokens) were nearly 2 times that of CoT (1044 tokens) and almost 4 times that of vanilla (560 tokens). This highlights the significant computational overhead required for multipersona ensemble reasoning.

The analysis also revealed a clear difference between the 2 evaluation modalities. As shown in both the average counts (Figure 5B) and the distributions (Figure 6B), interactions in the open-ended modality were generally more token-intensive. More importantly, the box plots in Figure 6B show that these open-ended interactions exhibited significantly greater variance (larger IQRs and more outliers) compared to their multiple-choice counterparts (Figure 6A). This indicates that open-ended questions not only require a higher average total token usage but also lead to less predictable and more variable costs.

In terms of efficiency, the family of RAG strategies demonstrated a superior balance of high accuracy and token economy. Crucially, our analysis accounts for the full cost, including retrieved context in the input tokens. While achieving high accuracy (as shown in Figure 2), their average token counts (Figure 5) remained significantly lower than RoT and often comparable to CoT. This highlights a fundamental difference in cost structure: the expense of RoT is primarily in the generated output (ie, computational reasoning), whereas a significant portion of RAG’s cost is front-loaded into the input prompt (ie, providing context). Furthermore, the RAG methods offer a spectrum of cost-performance options: hybrid RAG, which integrates CoT reasoning, logically incurs a higher token cost than basic or advanced RAG, positioning it as a premium option for when maximal accuracy is required. This positions the RAG framework as a strategically efficient approach for deploying reliable clinical AI systems. It allows developers to balance computational cost and performance requirements by selecting the appropriate RAG configuration—from token-frugal methods for simpler tasks to more powerful, context-rich options such as hybrid RAG for high-stakes clinical decisions.

To investigate the influence of training data currency on model performance, we conducted an analysis on the “advanced” tier using GPT-4o, comparing its performance on questions from precutoff vs postcutoff sources (Figure 7).

For strategies relying on internal knowledge (vanilla, CoT, and RoT), performance was consistently higher on the precutoff data (eg, 83.68% vs 76.32% for vanilla). This indicates a dependency on the model’s training history for these non-RAG methods.

The analysis of RAG-based strategies revealed that performance is critically dependent on knowledge source alignment. A striking example is basic-RAG, where providing out-of-domain guideline knowledge to answer advanced questions led to a sharp performance drop on postcutoff data (55.26%), the lowest of any strategy. This highlights the risk that a misaligned RAG can be more detrimental than no RAG at all.

In contrast, when the knowledge source was correctly aligned, RAG’s effectiveness became evident. The advanced RAG strategy achieved the highest accuracy on postcutoff questions (97.37%), demonstrating the model’s strong ability to reason over novel information when the correct context is supplied. Most notably, hybrid RAG maintained high and stable accuracy across both precutoff (91.32%) and postcutoff (89.47%) data, indicating its robustness.

These findings confirm that while non-RAG strategies are sensitive to training data currency, a RAG architecture’s reliability is primarily determined by its knowledge base. A comprehensive and well-aligned knowledge source, as demonstrated by a hybrid RAG, allows the system to perform reliably, independent of the model’s internal knowledge cutoff.

The increasing application of LLMs in medicine raises a pivotal question regarding their reliability in knowledge-intensive subspecialties such as MSI cancers, a domain where no gold-standard evaluation benchmark existed at the time of our analysis. To address this challenge, we developed MSIC-Bench. This benchmark features a two-tiered structure based on clinical guidelines (eg, NCCN and ESMO) and a curated, evidence-based knowledge base (MSICKB), allowing for separate assessment of an LLM’s performance on both consensus and nuanced scientific knowledge. Using this framework, we evaluated 3 leading LLMs across 4 prompting strategies, analyzing their accuracy, safety, error composition, and token usage.

Our findings reveal a significant “scaffolding effect,” where selection-based formats inflate accuracy compared to more realistic open-ended scenarios. This performance drop highlighted specific failure modes, notably in chemotherapy-related decisions. To address these failures, RAG proved to be a critical intervention. It not only substantially improves accuracy but also fundamentally shifts the error profile from high-risk “fabrications” to safer “I don’t know” responses. Finally, our analysis highlights key complexities for real-world applications: model-specific vulnerabilities, the importance of knowledge-to-task alignment in RAG, and the high token usage of certain prompting strategies.

Our evaluation of MSIC-Bench yields 3 principal insights. First, we identified a pronounced “scaffolding effect”: LLMs perform substantially better on multiple-choice than on open-ended questions, indicating that widely used multiple-choice benchmarks likely overestimate their real-world clinical capabilities and that honest behavior cannot be assessed unless the task design explicitly allows refusals. Second, error composition analysis shows that in this specialized domain, the dominant failure mode of non-RAG strategies is internal knowledge deficit, meaning that better reasoning prompts alone cannot compensate for missing domain knowledge; with RAG, this bottleneck shifts from “knowing” to “finding,” as retrieval failure becomes the primary source of error. Third, RAG emerges as the key intervention for deploying LLMs in MSI oncology, with performance and safety tightly coupled to knowledge-base quality: domain-aligned and comprehensive resources (as in advanced and hybrid RAG) markedly improve accuracy and convert high-risk fabrications into safer refusals, but also introduce new dependencies on retrieval precision and computational cost, making optimization of the retrieval module and careful curation of knowledge sources (eg, through the use of structured methodologies such as ontologies [35,36] and curated knowledge bases [14,37,38]) central to future clinical AI development.

Our findings are consistent with and extend the growing body of literature on LLM evaluation in specialized domains. For instance, the performance gap we identified between multiple-choice and open-ended modalities (the “scaffolding effect”) aligns with similar discrepancies observed by Song et al [13] in their study on RAG, confirming that evaluation formats significantly impact perceived model capabilities in clinical contexts.

Furthermore, our findings contribute to the continuing discussion about the roles of reasoning vs knowledge in LLM performance. Our conclusion regarding the limited efficacy of reasoning prompts such as CoT in knowledge-intensive tasks is corroborated by prior work. For instance, prior work [39] similarly found that applying CoT in the knowledge-driven domain of rare disease diagnosis could paradoxically decrease accuracy. Our study provides a clear mechanistic explanation for this phenomenon. By using error composition analysis, we demonstrate that when foundational knowledge is absent—as is often the case in specialized medicine—the primary failure mode is internal knowledge deficit. In such scenarios, structured reasoning prompts cannot compensate for the core knowledge void.

Crucially, our work extends this understanding by showing how RAG transforms the problem. We demonstrate that by supplying external knowledge, RAG effectively eliminates internal knowledge deficit, but in doing so, it shifts the primary bottleneck to retrieval failure. This finding refines the distinction between reasoning-intensive and knowledge-intensive tasks, suggesting that for RAG-based systems, retrieval-intensity emerges as a new, critical dimension for analysis and optimization.

Our study has several limitations that also suggest directions for future work. First, MSIC-Bench is limited to a single clinical problem and reflects only a static snapshot of knowledge related to MSI-associated cancers, which restricts its breadth of coverage. Future work should broaden and update the benchmark within MSI cancers by adding more diverse questions, incorporating real-world case-based scenarios, and regularly revising content to reflect evolving evidence and model capabilities. Second, the benchmark is text-only, whereas real-world decision-making is multimodal. An important next step is the development of benchmarks that integrate images, genomic data, and electronic health records. Third, our question-answering format simplifies clinical reasoning, which is often iterative and conversational; future evaluations should include more realistic case-based and dialogue-style tasks.

Fourth, our study used a foundational RAG implementation. Our error analysis identified retrieval failure as the new primary bottleneck, which points to a clear and critical direction for future work. Research should prioritize the investigation of more sophisticated RAG techniques, such as iterative retrieval and query rewriting, to directly address this retrieval challenge. Fifth, our initial RAG evaluation used a “split-index” (oracle routing) strategy. While this design allowed us to isolate the impact of knowledge source quality, we acknowledge that it does not fully reflect real-world scenarios. Our introduction of the hybrid-RAG model was a first step to address this, but future work should focus on developing more sophisticated dynamic routing mechanisms. Sixth, the same static prompt template was applied across all RAG settings. In the advanced and hybrid RAG strategies, the phrase “clinical guideline context” may not have fully matched the nature of the retrieved MSICKB-derived evidence, which could have influenced how the model interpreted the provided context. Future work should explore source-aware prompt designs that better align the instruction wording with the type of retrieved evidence. Seventh, our analysis revealed a critical trade-off between safety and utility in RAG systems. While RAG effectively curbed dangerous “fabrications,” it also introduced “false refusals”—a failure to answer otherwise answerable questions that limits clinical utility. This highlights that the goal for future research is not simply to maximize refusals for safety, but to find an optimal balance. Future work should therefore focus on strategies to minimize false refusals without reintroducing a tendency to fabricate.

Eighth, our analysis of parametric memory was constrained by model cutoff dates, preventing a direct before-and-after comparison for Claude and Gemini. However, our conclusion that “seeing is not knowing” is supported by strong indirect evidence: our error composition analysis (Figure 4) shows that internal knowledge deficit remained the dominant non-RAG failure mode for all models. This finding is consistent with multiple studies showing that LLMs often fail to reliably recall information, even when it is present in their training data, a phenomenon particularly pronounced in specialized, long-tail knowledge domains such as MSI cancers [40,41]. Ninth, as the MSICKB knowledge base and the “advanced” tier questions were developed by the same research group, a discussion of potential circular validation is warranted. To mitigate this risk, every ground truth answer is strictly grounded in and explicitly linked to its source peer-reviewed publication via a PubMed ID. However, more robust external validation remains an important direction for our future work. Our primary planned next step is to collect and annotate real-world clinical data to create an independent test set for validating our benchmark and retrieval framework. We also note that this specific challenge of circular validation, where the retrieval corpus overlaps with the benchmark’s source material, is a widely recognized issue in the evaluation of specialized RAG systems [42,43]. Additionally, our token usage estimates relied on a single tokenizer (tiktoken), which only approximates tokenization behavior for non-OpenAI models. Future work should use model-specific tokenizers to obtain more accurate cost estimates. Finally, because LLMs evolve rapidly, our results capture only a single time point, emphasizing the need for continuous and longitudinal benchmarking in clinical AI.

In this study, we addressed the critical absence of a standardized evaluation tool for LLMs in MSI cancers by developing and applying MSIC-Bench. Our benchmark revealed that the primary limiter of current LLMs is a profound deficit in domain-specific knowledge. Our results demonstrate that a well-designed RAG architecture is the pivotal intervention to address this knowledge gap. However, it is not a simple fix. We found that RAG transforms the system’s failure modes, shifting the primary bottleneck from knowledge deficits to retrieval failures. This insight establishes a core principle for the development of trustworthy clinical AI: the quality and comprehensiveness of the knowledge base and the precision of the retrieval system are the most critical components. Ultimately, enhancing clinical RAG systems requires a dual focus on optimizing retrieval mechanisms and improving the model’s contextual reasoning. MSIC-Bench provides not only a comprehensive capability profile of LLMs in this domain but also a replicable methodology to guide this future development, ensuring that knowledge-augmented AI assistants can be validated and safely integrated into specialized medical fields.