In recent years, large language models (LLMs) have undergone rapid development.

Since the introduction of OpenAI’s GPT-3 [1], both industry and academia have actively competed to develop powerful LLMs [2-9], which are now widely adopted in daily life. More recently, models that integrate data from multiple modalities, particularly vision-language models (VLMs) [10-16], have garnered significant attention among multimodal approaches.

A common trend in developing these models has been to scale up both model size and training data in pursuit of improved performance. However, the high demand for computational resources and massive datasets presents substantial barriers to broader accessibility and development. Moreover, while training-time scaling laws have led to certain improvements, many models still struggle with complex reasoning tasks.

To address these challenges, test-time scaling has recently emerged as an effective method for enhancing LLM performance during inference [17-20]. For instance, rather than generating an immediate final prediction, a model can be configured to produce intermediate step-by-step logical deductions or generate multiple candidate responses to find a reliable consensus. Without requiring additional training or fine-tuning, test-time scaling improves both reliability and accuracy, especially on tasks that require multistep reasoning. By increasing the token budget or generating multiple candidate responses, test-time scaling enables models to produce more detailed and structured chain-of-thought reasoning. This approach is particularly synergistic with reasoning-optimized models, which are trained using supervised fine-tuning (SFT) on chain-of-thought–annotated datasets or reinforcement learning (RL) with specialized objectives [21-23]. The combination of reasoning models and test-time scaling has shown state-of-the-art results on complex tasks such as mathematics problem solving and program code synthesis [6,9].

Recently, applications of LLMs and VLMs in the medical domain have also gained momentum [24-30]. Alongside these applications, both reasoning models [29-33] and test-time scaling [30,34,35] are being increasingly adopted in medical AI research. Recent studies demonstrate that the use of reasoning models and test-time scaling significantly boosts performance on medical benchmark datasets [30].

Despite growing interest in test-time scaling in the medical domain, many important aspects remain underexplored. Although various test-time scaling strategies have been proposed, only a limited subset has been systematically evaluated in medical applications. For example, it remains unclear whether shorter or longer reasoning is more advantageous in the medical domain, or whether sequential or parallel scaling is more effective. Furthermore, despite differences in training data and methods across models, many prior studies have applied test-time scaling strategies uniformly, without accounting for each model’s unique characteristics. Given the wide range of difficulty in medical tasks, it is also important to examine whether reasoning via test-time scaling is equally beneficial across different task types. In addition, with the increasing relevance of VLMs in clinical and diagnostic contexts, investigating test-time scaling strategies for VLMs in the medical domain is essential.

Crucially, while foundational studies have established test-time scaling paradigms in the general domain by focusing primarily on text-based logic such as mathematics and coding [18-20], their findings cannot readily predict the multilayered complexities inherent to clinical AI. General-domain frameworks fail to capture the domain-specific bottlenecks and unique behavioral shifts that emerge in medical environments. Specifically, it remains unexamined how test-time compute interacts with the intricate visual reasoning required for complex medical images, whether specialized medical knowledge fine-tuning paradoxically compromises a model’s baseline numerical reasoning in clinical calculations, and how scaling affects a model’s cognitive vulnerability when confronted with incorrect information embedded in user prompts. Resolving these questions is vital to discovering insights unique to medicine that transcend mere empirical replication of general-domain findings.

To bridge these gaps, in this paper, we present a comprehensive investigation of test-time scaling for medical applications. Figure 1 illustrates the overall framework of our study. First, we analyze how increasing the token budget affects the performance of various LLMs and VLMs across multiple medical benchmark datasets. We explore how this performance varies with factors such as model size, model characteristics, and task complexity. Next, we compare sequential and parallel test-time scaling strategies, highlighting their relative effectiveness in medical tasks. Finally, we evaluate the robustness of test-time scaling under user-driven factors, such as misleading contextual information embedded in prompts. Ultimately, the primary aim of this study is to establish a domain-specific framework for scaling inference compute in medical AI. We hypothesize that the optimal scaling strategy is fundamentally dependent on both the inherent reasoning capacity of the model and the specific complexity of the clinical task, and that appropriate scaling can significantly improve model robustness against misleading user inputs.

In our experiments, we evaluate LLMs on five medical benchmark datasets, including four medical question answering (QA) datasets and one medical calculation dataset. Characteristics of all medical benchmark datasets used in this study are summarized in Table 1.

PubMedQA [40] is a medical QA dataset comprising 500 research questions, each paired with a relevant abstract. The model must answer each question with one of three choices: “yes,” “no,” or “maybe.”

MedQA [41] includes medical QA pairs derived from textbooks. In our study, we use a subset of 1273 multiple-choice questions based on the USMLE (United States Medical Licensing Examination), each with five answer options.

MedBullets [42] is another USMLE-style medical QA dataset, consisting of 308 multiple-choice questions, each with five options, similar in format to MedQA.

MedXpertQA (text) [43] is a recently proposed medical QA benchmark that includes both textual and multimodal (text and image) questions. For our LLM experiments, we use only the text-based subset, which contains 2450 questions with ten answer choices each.

MedCalc-Bench [44] is a medical calculation benchmark comprising 1047 questions, each accompanied by a corresponding patient note. The dataset is designed to evaluate models’ ability to perform clinical calculations based on contextual patient information.

We define the difficulty of the medical QA datasets in ascending order as follows: PubMedQA, MedQA, MedBullets, and MedXpertQA, based on previous studies [34,35,40-45], the number of choices, and the level of reasoning required to answer the questions. Additionally, given that prior studies have demonstrated the substantial reasoning demands involved in calculations [17-20], we consider MedCalc-Bench a difficult dataset that requires extensive reasoning.

Next, we evaluate VLMs on two medical multimodal datasets.

OmniMedVQA [46] is a large-scale benchmark designed for evaluating medical VLMs. OmniMedQVA consists of simple and short questions based on provided images, making its overall difficulty relatively easy. For our experiments, we randomly sample 5000 questions spanning three categories that require relatively more reasoning: disease diagnosis, lesion grading, and other biological attributes.

Regarding MedXpertQA (multimodal), for our VLM experiments, we use a multimodal subset of the MedXpertQA dataset, which includes 2000 multiple-choice questions, each with five answer options. Each question is accompanied by one to six associated medical images. As the models have to process multiple images simultaneously and comprehend the contextual information in the questions, we consider MedXpertQA a challenging task.

We use the following LLMs for our experiments.

Regarding general instruction-tuned LLMs, we evaluate instruction-tuned versions of Llama 3 [5] (Llama 3-3B, 8B, 70B) and Qwen2.5 [8] (Qwen2.5-3B, 7B, 32B, 72B) as representative general-purpose LLMs. Both models have been fine-tuned on instruction-following datasets. Llama 3 is trained using SFT followed by RL from human feedback [47], whereas Qwen2.5 is fine-tuned using a combination of SFT, direct preference optimization [22], and group relative policy optimization [23]. In addition, we evaluate the Qwen3-Instruct models (Qwen3-8B, 30B-IT), which represent the updated nonthinking mode of the Qwen3 model [48].

Regarding general reasoning LLMs, we also evaluate the Qwen3-Thinking models (Qwen3-8B, 30B-TH), which are scaled to enhance the thinking capability of the Qwen3 models, specifically targeting improved quality and depth of reasoning [48]. Next, we include distilled versions of DeepSeek-R1 [9] (DeepSeek-R1-7B, 8B, 32B, 70B) as representative models optimized for general reasoning capabilities. While the original DeepSeek-R1 is trained using both SFT and RL, the distilled versions are trained solely via SFT using a reasoning dataset generated by the original DeepSeek-R1 model. Finally, we include the gpt-oss models [49] (gpt-oss-20B, 120B), OpenAI’s open-source series trained via SFT and RL.

Regarding medical LLMs, we evaluate LLMs specifically trained for the medical domain.

UltraMedical [27] is a medical LLM fine-tuned using SFT followed by preference optimization techniques such as direct preference optimization [22] or Kahneman-Tversky optimization [50]. We use the 8B and 70B variants of UltraMedical in our experiments (UltraMedical-8B, 70B).

Regarding medical reasoning LLMs, we also evaluate LLMs specifically designed for medical reasoning tasks. HuatuoGPT-o1 [29] (HuatuoGPT-7B, 8B, 70B, 72B) is a medical LLM trained on synthetic medical problems featuring complex chain-of-thought reasoning. The model is trained using SFT and RL via proximal policy optimization [21].

In addition, we include m1 [34], a medical reasoning model trained with curated chain-of-thought–style data using SFT, but without RL. In our experiments, we use versions of m1 that are trained on a dataset of 1000 samples, which are denoted as m1-7B, 32B in this paper.

Regarding general VLMs, we evaluate five general-purpose VLMs: Llama 3-Vision [5] (Llama 3-Vision-11B, 90B), Qwen2.5-VL [51] (Qwen2.5-VL-7B, 32B), Qwen3-VL [52] (Qwen3-VL-8B, 30B-IT), Gemma 3 [53] (Gemma 3-12B, 27B), and LLaVA [13] (Large Language and Vision Assistant; LLaVA-7B, 13B). For Llama, Qwen, and Gemma models, we use instruction-tuned versions.

Regarding general reasoning VLMs, we also include general-domain models explicitly trained for reasoning tasks. First, the thinking version of Qwen3-VL [52] (Qwen3-VL-8B, 30B-TH) is evaluated. Next, LLaVA-CoT [54] (Large Language and Vision Assistant–Chain-of-Thought; LLaVA-CoT-11B) is a VLM fine-tuned with synthetic chain-of-thought data using SFT. Finally, we evaluate the preview version of QVQ [55] (Qwen With Vision and Questions; QVQ-72B), a large-scale multimodal reasoning model built upon Qwen2-VL-72B [15].

Regarding medical VLMs, we evaluate VLMs specifically tuned for the medical domain. MedGemma [39] (MedGemma-4B, 27B) is a medical variant of Gemma 3 trained on medical text and images. In our experiment, we use the instruction-tuned version of MedGemma. HuatuoGPT-Vision [38] (HuatuoGPT-Vision-7B, 34B) is another medical VLM fine-tuned on medical VQA data.

Regarding medical reasoning VLMs, lastly, we include the medical reasoning VLM, QoQ-Med [31], in our evaluation. QoQ-Med is trained on the CLIMB (clinical large-scale integrative multimodal benchmark) [56], a comprehensive dataset that incorporates diverse types of medical data, including ECG (electrocardiogram; 1D), chest X-rays (2D), and magnetic resonance imaging scans (3D). To enhance multimodal reasoning capabilities, the authors of a study [31] proposed a domain-aware group relative policy optimization, which applies hierarchical scaling strategies based on the domain of the input data. In our experiments, we evaluate two versions of QoQ-Med: QoQ-Med-7B and QoQ-Med-32B.

Table 2 summarizes the models used in our experiments and their key characteristics. In our experiments, we use a 4-bit quantized version of the models when the number of model parameters is 70B or more.

This study does not involve human participants, animal participants, medical records, or any personally identifiable information. All experiments were conducted exclusively using publicly available and anonymized medical benchmark datasets to evaluate LLMs. Therefore, this study does not constitute human participants research and did not require formal review or approval by an institutional review board or a local ethics committee, in accordance with standard institutional and national policies regarding the secondary analysis of publicly available data.

In previous experiments, we observed that test-time scaling, by simply increasing the token budget, is ineffective for many models. Except for certain reasoning models, most do not use the full available token budget during inference. To address this limitation, we investigate alternative scaling strategies, iterative sequential scaling and parallel scaling, as described in the Methods section. In this experiment, we evaluate two medical reasoning LLMs: HuatuoGPT-o1-7B (denoted as HuatuoGPT-7B) and m1-7B-1k (denoted as m1-7B).

Figure 6 presents a comparison between sequential and parallel test-time scaling across multiple medical benchmarks. On PubMedQA, the easiest QA task, both models show a decline in accuracy as the number of sequential sampling steps increases. Moreover, the accuracy achieved with parallel scaling is higher than that obtained by simply increasing the token budget. This suggests that additional iterations may not be beneficial for simpler tasks; in fact, excessive revisions can lead the models to deviate from initially correct answers. In particular, for m1, accuracy decreases even though coverage increases with more sequential steps, indicating that while the model may identify the correct answer during sequential scaling, unnecessary revisions can lead to an incorrect final output.

For MedQA and MedBullets, HuatuoGPT-o1 maintains the same trend, showing limited improvement from either sequential scaling or increased token budgets. In contrast, m1 achieves its highest accuracy when the token budget is evenly divided between sequential and parallel sampling (1:1 ratio), with performance comparable to that of simple token budget expansion. These findings suggest that for tasks of moderate difficulty, sequential scaling may offer little advantage for models that naturally generate shorter outputs. For these models, the performance gains from parallel scaling on simpler tasks are driven primarily by reasoning path diversity and answer consistency through majority voting, rather than the brevity of the reasoning itself. In comparison, models such as m1, which rely on deeper reasoning, benefit more from a balanced approach that combines diverse generation with iterative refinement.

On the more challenging MedXpertQA benchmark, which demands complex reasoning, HuatuoGPT-o1 shows relatively consistent performance across all scaling strategies, including increased token budget, sequential scaling, and parallel scaling. In contrast, m1 achieves its highest accuracy and coverage with fully sequential scaling, slightly surpassing its performance under the increased token budget condition.

On MedCalc-Bench, which also involves complex reasoning but with distinct characteristics, both models show improved performance as the number of sequential samples increases. However, the highest accuracy is attained when applying test-time scaling via a larger token budget. These findings suggest that step-by-step sequential refinement benefits reasoning-intensive question-answering tasks, while generating a single, extended reasoning path is more effective for tasks requiring precise calculations.

In summary, under constrained token budgets, parallel scaling proves highly effective for simpler tasks. This advantage stems from the model exploring a diverse set of reasoning paths and arriving at a reliable conclusion through answer consistency. In contrast, for models that rely on more extensive reasoning, sequential scaling or increasing the token budget becomes increasingly advantageous as task complexity grows.

In addition to evaluating task performance, we analyzed the computational cost, specifically the inference latency (wall-clock time), associated with each scaling strategy to assess their viability for clinical deployment. As illustrated in the bottom row of Figure 6, increasing the proportion of sequential reasoning results in a substantial increase in processing time per sample. This delay occurs because sequential scaling relies on iterative generation, where each step must wait for the previous one to complete, while the context length continuously expands. Conversely, heavily parallelized configurations maintain significantly lower latency. By using batched inference, parallel scaling processes multiple samples simultaneously without a proportional increase in wall-clock time. This quantitative analysis reveals a critical cost-performance trade-off, indicating that parallel scaling is highly efficient for time-sensitive clinical applications.

Finally, we investigate the impact of test-time scaling on robustness to user-driven factors. Again, we evaluate two medical LLMs, HuatuoGPT-o1-7B and m1-7B-1k, across two types of test-time scaling strategies. The first strategy simply increases the token budget by a factor of 16 from the default setting, as described in the first experiment. The second strategy combines iterative sequential scaling and parallel scaling using the optimal configuration identified in the second experiment. Figure 7 shows the accuracy of both models under various user-driven perturbations and test-time scaling strategies.

First, Figure 7A shows that user-driven perturbation generally degrades the performance of HuatuoGPT-o1 when no test-time scaling strategy is applied. When test-time scaling is applied by simply increasing the token budget, the improvement is minimal, and in some cases, performance even declines. In contrast, applying the optimal test-time scaling strategy significantly improves robustness against user-driven factors in most scenarios. The benefit is particularly pronounced when the task is easier, such as in MedQA. This is likely because HuatuoGPT-o1 tends to generate relatively short responses by default, and merely increasing the token budget does not effectively encourage the model to use the additional tokens for deeper reasoning.

On the other hand, m1 also exhibits degraded performance in the presence of user-driven factors, as illustrated in Figure 7B. However, unlike HuatuoGPT-o1, increasing the token budget proves more effective than applying the optimal sequential-parallel scaling strategy for the m1 model. This is due to m1’s inherent tendency to generate long chain-of-thought reasoning when solving problems. As sequential-parallel scaling produces shorter responses at each iteration, it may hinder m1 from fully developing its reasoning and reaching a well-considered conclusion.

In summary, model behavior varies depending on the chosen test-time scaling strategy when user-driven factors exist. This highlights the importance of selecting an appropriate test-time scaling method tailored to each model type to enhance robustness against such perturbations. Nonetheless, some common patterns emerge across models. For relatively easy tasks (eg, MedQA), both models can recover performance comparable to that without user-driven interference through test-time scaling. In contrast, for more challenging tasks (eg, MedXpertQA), the performance gap between conditions with and without user-driven factors remains substantial, even when test-time scaling is applied. Notably, models are particularly sensitive to the expertise level of the physician providing the additional input. This suggests that as task complexity increases, LLMs may struggle to initiate reasoning with high confidence, leading them to heavily rely on the initial context or cues provided by the user. Consequently, when such cues are misleading, the model is more likely to follow an incorrect reasoning path and, due to the increased complexity, has greater difficulty recovering and arriving at the correct answer. These findings emphasize the amplified impact of user-driven perturbations in difficult scenarios and highlight the importance of minimizing such perturbations by providing only neutral and essential information when prompting LLMs for complex or high-stakes medical QA tasks.

To contextualize our findings, the primary aim of this study was to establish a domain-specific framework for scaling inference compute in medical AI. We hypothesized that the optimal scaling strategy depends heavily on the reasoning capacity of the model and the complexity of the clinical task. Our overall results broadly confirm these hypotheses. We demonstrated that simply increasing the computational budget during inference is not a universal remedy. Instead, the efficacy of test-time scaling is highly conditional, varying significantly across model architectures, task difficulties, and clinical contexts.

First, we found that simply increasing the token budget is not a universal solution for enhancing medical LLM or VLM performance. A distinct contrast exists between nonreasoning and reasoning models. Nonreasoning models (eg, Llama 3 and Qwen2.5) exhibited performance saturation, using fewer than 1000 tokens even when provided with larger budgets, resulting in negligible accuracy gains. In contrast, reasoning-optimized models demonstrated adaptive token usage. Notably, the gpt-oss models exhibited behavior aligned with “thinking-optimal” scaling. They maintained efficient, low token usage for simpler tasks such as PubMedQA but increased consumption for complex benchmarks such as MedXpertQA, which correlated directly with accuracy improvements. Furthermore, the efficiency of this scaling appears to be heavily influenced by the nature of the model’s training data. The m1 model, trained on DeepSeek-R1 traces, used more tokens for deep reasoning than HuatuoGPT-o1, which relied on GPT-4o traces.

Second, test-time scaling for VLMs currently lags behind that of LLMs. Most VLMs failed to leverage increased budgets effectively and showed limited improvements in reasoning depth. Exceptions to this trend were observed in Qwen3-VL variants and MedGemma-27B. This suggests that advanced posttraining methods such as Soft Adaptive Policy Optimization and larger parameter sizes are critical enablers for multimodal test-time scaling. Specifically, we observed that scaling model parameters in conjunction with medical domain training proves beneficial.

Third, we demonstrated that the optimal scaling strategy is highly task-dependent. For simpler tasks, excessively long reasoning paths are not necessarily beneficial. Iterative sequential scaling proved harmful on easier datasets such as PubMedQA. This performance drop is likely due to compounding errors during unnecessarily extended reasoning. In this case, parallel scaling using majority voting yielded superior results. This success arises from exploring a diverse set of reasoning paths and aggregating them through majority voting to ensure answer consistency. Conversely, reasoning-intensive tasks such as MedXpertQA and calculation tasks such as MedCalc-Bench benefited from extended sequential reasoning.

Finally, while test-time scaling enhances robustness against use-driven perturbations, it requires model-specific tailoring. We observed that scaling strategies generally mitigated the negative impact of misleading expert hints but often failed to fully restore baseline performance on difficult tasks. Crucially, the effective strategy varied by model architecture. HuatuoGPT-o1 benefited most from a hybrid sequential-parallel approach. However, m1 relies on generating long, uninterrupted reasoning chains and performs best with a simple expansion of the token budget. This underscores the necessity of aligning scaling strategies with both the difficulty of the clinical task and the intrinsic reasoning behaviors of the specific model being deployed.

Prior studies on test-time scaling in general domains, such as mathematics and coding, have established that increasing inference compute can improve reasoning, though the relationship is not strictly monotonic. These works demonstrate that optimal reasoning lengths exist and that excessive self-revision can sometimes degrade performance [18-20]. While these foundational works highlight the need for task-dependent scaling strategies, our study reveals that in the high-stakes medical domain, the limitations of test-time scaling go beyond mere overthinking and manifest through three unique, domain-specific bottlenecks.

First, text-centric scaling limits encounter a fundamentally different bottleneck in multimodal clinical environments. While prior work focuses on finding the optimal token length to balance reasoning depth and diversity, our evaluation of VLMs shows that the issue is not about optimal length but rather a structural inability to integrate complex visual clues. Simple token expansion fails to trigger meaningful multimodal reasoning, indicating that safe medical AI deployment requires fundamental vision-language alignment improvements rather than just optimizing computational scaling.

Second, we identify a domain-specific disparity in scaling efficiency between qualitative clinical and quantitative procedural logic. Unlike general studies that evaluate scalability based on parameter size or general reasoning datasets, we demonstrate that medically fine-tuned LLMs excel in text-based medical QA but exhibit degraded scaling efficiency on clinical calculation tasks compared to general-domain models. This disparity does not necessarily imply that medical fine-tuning inherently impairs reasoning capabilities. Instead, it suggests that current medical alignment heavily prioritizes qualitative clinical knowledge over the general numerical and procedural reasoning captured by calculation benchmarks. Consequently, test-time scaling strategies behave inconsistently depending on whether the clinical task relies on deep domain-specific expertise or general numerical computation, an insight unique to the medical domain.

Third, our work uncovers a unique cognitive vulnerability to perceived clinical authority. Standard robustness evaluations in prior test-time scaling typically focus on a model’s ability to self-correct against random noise or generic misleading logic. In contrast, we demonstrate that when models are confronted with deceptive hints framed specifically as expert physician opinions, they readily abandon their correct reasoning pathways. Under this hierarchical clinical pressure, even optimal scaling strategies fail to restore baseline performance, which represents a distinct vulnerability that differentiates our work from general-domain robustness findings.

While our experimental design controls for the token budget to evaluate the cognitive limits of reasoning length, it is important to acknowledge that token count does not equate to equal computational cost across different model architectures. Generating the same number of tokens with a large-scale model consumes substantially more compute (floating point operations per second), energy, and financial resources than a smaller model. Therefore, evaluating deployment feasibility requires balancing reasoning performance against strict computational constraints.

Furthermore, different test-time scaling strategies exhibit distinct cost-performance trade-offs. As observed in our experiments, parallel scaling can be implemented efficiently using batched inference, where model weights are loaded only once. This minimizes the increase in GPU memory footprint to just the required key-value cache and significantly reduces wall-clock time, making it highly suitable for time-sensitive environments such as emergency departments. Conversely, sequential scaling inherently increases latency due to its serial nature and expanding context length, limiting its use to nonurgent, complex diagnostic scenarios. Ultimately, health care institutions must carefully weigh these factors. They should choose whether to deploy a massive model or to apply efficient parallel scaling on a smaller, medically aligned model based on their specific latency requirements and computational budgets.

Although we conducted a broad investigation of test-time scaling in the medical domain, several limitations remain. First, our study focused on scaling strategies such as increasing the token budget, iterative self-revision via sequential scaling, and generating multiple responses in parallel. However, other promising strategies, such as using trained verifiers to select the most accurate response among candidates, were not explored. In addition, compared to LLMs, VLMs require more comprehensive evaluation under test-time scaling. While our results show that test-time scaling can improve robustness to user-driven factors, it was not sufficient to fully restore performance to baseline levels observed without such perturbations. Future work should investigate more advanced or hybrid test-time scaling techniques that can further mitigate the impact of user-driven factors. Finally, our experiments were limited to open-source LLMs, while proprietary models such as GPT (OpenAI), Claude (Anthropic, PBC), or Gemini (Google LLC) were not included. Extending test-time scaling evaluations to these models represents an important direction for future research in the medical domain.

A notable limitation of our study is that the evaluation relies on specific benchmark datasets, which serve as finite samples of the much broader population of real-world medical questions. Consequently, the reported accuracy metrics carry inherent statistical uncertainty. In scenarios where performance differences across models or scaling strategies are marginal, identifying a single optimal model based solely on empirical accuracy may not fully reflect their true generalization ability. Future research evaluating LLMs in health care should incorporate rigorous statistical frameworks to quantify this uncertainty. Using methodologies to construct confidence sets of potential best performers [61-63] would provide a more robust and statistically sound approach to model selection when a clear single winner is not evident.

Additionally, as with most current LLM evaluations, we cannot completely rule out the possibility of data contamination, where public benchmark questions might have been included in the base models’ pretraining corpora. However, the core focus of this study is the relative performance gain achieved through test-time scaling rather than absolute baseline accuracy. If models were merely retrieving memorized answers, increasing the token budget or applying sequential scaling would not yield the significant accuracy improvements we observed. Furthermore, our perturbation experiments using user-driven factors demonstrate that models engage in active, dynamic reasoning rather than static memory retrieval. Nonetheless, future studies using completely private or recently curated clinical datasets will be essential to further validate the extent of test-time scaling benefits free from any potential memorization bias.

In this paper, we conducted a comprehensive investigation of test-time scaling in the medical domain across various types of LLMs and VLMs. Our experiments demonstrate that longer reasoning is not universally beneficial across all medical tasks. For simpler tasks, extensive reasoning is unnecessary, and models that produce concise reasoning traces offer better computational efficiency. In contrast, for complex and challenging tasks, generating longer chain-of-thought reasoning leads to better performance, with models capable of producing extended reasoning traces outperforming their counterparts. Additionally, we observed that medical LLMs tend to perform worse on medical calculation tasks compared to general domain LLMs. This trend likely reflects that current medical fine-tuning practices heavily focus on qualitative textual medical knowledge, meaning these models are less optimized for the procedural, calculation-heavy demands captured by such benchmarks rather than enduring an actual impairment of reasoning capabilities.

Furthermore, we examined whether sequential or parallel test-time scaling provides greater benefits. Our findings indicate that the optimal test-time scaling strategy varies depending on both the model type and the difficulty of the task. Additionally, we found that selecting an appropriate scaling strategy is crucial for enhancing model robustness against user-driven factors, such as misleading or biased information in the prompts. Notably, the impact of these user-driven factors becomes more pronounced as task difficulty increases, emphasizing the importance of minimizing perturbations by providing only neutral and essential information for complex tasks. A summary of our findings is presented in Tables 3 and 4.

Beyond benchmark performance, these findings hold broader implications for the real-world deployment of medical AI. As health care institutions increasingly integrate AI into clinical workflows, treating inference compute as a static resource is no longer viable or safe. Our study highlights that ensuring patient safety, diagnostic accuracy, and operational efficiency requires a dynamic approach to model deployment. Computational strategies must be carefully tailored to the specific urgency and complexity of the medical scenario. Ultimately, understanding when and how to appropriately scale reasoning at test time is a crucial foundational step toward developing clinical AI systems that are not only computationally efficient but also robust, resilient, and trustworthy in high-stakes health care environments.