Due to the relatively different structure of semantic Q and A and the limited number of Q and A, we analyzed all of them separately.

Regarding numeric Q and A types and in the search for meaningful parameters that might influence LLM’s performance, the benchmark included Q and As based on three medical labels (medical subject type, medical discipline, and prevalence) and three nonmedical sublabels (Q and A types, question length, and answers distribution) as further detailed in Multimedia Appendix 3. All Q and As were randomly selected, and although the total number of Q and As per label varied, each label contains an identical number of selected Q and As per sublabeled entity, with no repetition across selections.

In this study, we used two state-of-the-art LLMs: GPT-4 (gpt-4‐0125-preview) and Claude 3 Opus (claude-3-opus-20240229). Both models’ parameters included temperature=0 and maximum tokens=300. All queries were sent to each LLM using its respective application programming interface. The application programming interface calls were made using R (version 4.2.2; Posit Software, PBC) via R Studio. Further descriptions of the prompts and suitable examples are presented in Multimedia Appendix 1

We evaluated LLM’s performance using the following metrics: (1) accuracy—for both semantic and numeric Q and As, the total number of correct answers suggested by the LLM divided by the total answers suggested by the LLM (excluding IDK answers); (2) answer rate (AR)—for both semantic and numeric Q and As, the total number of both correct and wrong answers suggested by the LLM (excluding IDK answers) divided by the total answers suggested by the LLM (including IDK answers); and (3) majority—for numeric Q and As only, the option that is selected as the correct answer most frequently among all given options in a questionnaire.

To test both the effects of adding IDK as a possible answer and changing the order of answers (including IDK), 8 different prompts were tested on 100 randomly selected numerical questions. Four prompts included IDK with a different order of possible answers, while four excluded IDK. To prevent bias in the selection process across specific question types, difficulty levels, medical disciplines, and question lengths, and to accurately represent the proportion of each question type in the EBMQA dataset, we randomly selected the questions included in the questionnaire.

To validate the Q and As in the EBMQA, 2 physicians and 4 clinical-year medical students (3 females and 3 males; aged between 28 and 35 years; all educated and licensed in Israel) answered the questionnaire. Each completed it first with the IDK option and then with mandatory guessing on their previous IDK responses. Their accuracies, with and without guessing, were compared to LLM’s performance.

All statistical analyses were performed using R Studio (R version 4.2.2). Categorical variables were represented as percentages, while continuous variables were represented as means and SDs for normally distributed data, or medians and IQRs otherwise. The cutoff for statistically significant results was set at α=.05, and 95% CIs were calculated. Proportions comparison was conducted using the “Proportion test.” Spearman correlation was used to analyze correlations between 2 quantitative variables.

This study was approved by the Tel Aviv University Ethics Committee (institutional review board protocol number 0008527‐2). All questionnaire data were anonymized and deidentified in accordance with HIPAA (Health Insurance Portability and Accountability Act) Safe Harbor privacy rules. Informed consent was received by all the participants. Before answering the questionnaire, participants were informed that the study was being conducted for research purposes only, no personal or sensitive data would be collected, answers would not be identified in the results, participation would be voluntary, and informed consent would be provided by answering the questionnaire. Appropriate measures were taken to ensure compliance with relevant privacy guidelines. No compensation was provided to participants.

The EBMQA contains 105,222 Q and As. In addition, each Q and A pair was labeled according to metadata labels and medical labels.

Of the 105,222 Q and As, a set of 24,542 questions was presented to each LLM. “Numeric” Q and As comprised 90% (22,000/24,542) of the questions, whereas “semantic” Q and As accounted for the remaining 10% (2542/24,254).

Both LLMs demonstrated better performances in the semantic Q and As than in the numeric Q and As in terms of accuracy (Claude 3: 68.65%, 1592.78/2320, vs 61.29%, 8583/14,005, P<.001; GPT-4: 68.38%, 1708.85/2499, vs 56.74%, 12,038/21,215, P<.001) and AR (Claude 3: 94.62%, 2320/2542, vs 63.66%, 14,005/22,000, P<.001; GPT-4: 98.31%, 2499/2542, vs 96.4%, 21,215/22,000, P<.001).

From an intermodel perspective, Claude 3 outperformed GPT-4 in numeric accuracy, though no significant difference was found in semantic accuracy. However, in comparison to Claude 3, GPT-4 had a higher AR in both semantic and numeric questions (Table 1). Focusing on numeric accuracy and excluding any questions that one or both LLMs responded to with IDK results in the exclusion of 8133 questions and a total of 13,867 answered questions. Keeping the same trend, Claude 3 outperformed GPT-4 in numeric accuracy (8491/13,867 vs 8255/13,867; P=.004).

Based on the results of the questionnaire, the average accuracy of Claude 3 with the IDK option versus without it was not significantly different (64.25%; mean 64.25, SD 3.95, vs 59.25%; mean 59.25, SD 5; P=.17). A similar trend was noted for GPT-4 (55.75%; mean 55.75, SD 1.71, vs 53.25%; mean 53.25, SD 2.89; P=.24). In addition, within each subgroup—Claude 3 with and without the IDK option, and GPT-4 with and without the IDK option—no single answer-option-order prompt was significantly superior to the others (Multimedia Appendices 7 and 8).

Claude 3 and GPT-4 achieved higher average accuracy rates, with or without the IDK option, than random guessing (33%, n=33) or majority guessing (47%, n=47). However, both models had lower average accuracy rates compared to humans with the IDK option (82.3%; mean 82.3, SD 2.82) or without it (78.2%; mean 78.2, SD 3.6; Multimedia Appendix 9).

The accuracy gap between the highest and lowest accuracy rates in each LLM was calculated, revealing a median difference of 7% (IQR 5%‐10%; Multimedia Appendix 10). Focusing on disorders selected sublabels, Claude 3 performed well in neoplastic disorders but struggled with genitourinary disorders (69%, 676/984 vs 58%, 464/803; P<.0001), while GPT-4 excelled in cardiovascular disorders but struggled with neoplastic disorders (60%, 1076/1783 vs 53%, 704/1316; P=.0002; Multimedia Appendix 11). Furthermore, among sublabel disorders queried over 200 times, Spearman correlations between Q and A and accuracy rate in both Claude 3 and GPT-4 were insignificant (ρ=0.12, P=.69; ρ=0.43, P=.13).

This study aimed to highlight the current gaps in the medical knowledge of LLMs and their current ability to surpass humans. We presented a method to create an EBMQA from a structured knowledge graph and benchmarked two state-of-the-art LLMs (GPT-4 and Claude 3) [16,17]. We demonstrated that both LLMs performed better in semantic Q and As than in numerical Q and As by asking more than 24,000 Q and As (Table 1). Claude 3 outperformed GPT-4 in numerical Q and As and showed similar results in semantic Q and As, although it exhibited significantly lower ARs (Table 1). A validation test indicated that the numerical accuracy rates of Claude 3 and GPT-4 were higher than majority guessing but remained lower than those of medical experts (Multimedia Appendix 12).

The use of knowledge graphs for evaluating LLMs is gaining popularity [20-22]. Kahun’s structured knowledge graph enabled us to generate both semantic and numeric labeled Q and A pairs, without using advanced models [22]. Our Q and A generation process, which relies on templates designed to fit a source-target-background graph structure, can apply to other graphs with a similar structure. In addition, this relatively large knowledge graph allowed us to create a massive EBM dataset. Moreover, we embraced a data-driven approach in which distractors were based on subanalysis distribution rather than specific or random values.

The EBMQA, which consists of 105,222 straightforward single-line Q and As, was designed to mimic physicians’ strategy of breaking complex medical scenarios into less complicated problems, unlike medical licensing examination datasets, which are typically complex-case oriented [14,23]. In addition, the EBMQA addresses numeric and semantic data, which is considered fundamental for physicians [24,25], while dealing with data from studies and embracing the EBM approach [7], as opposed to the abstracted-based yes or no or maybe Q and As in PubMedQA [15].

A major concern regarding applying LLMs in health care is the uncertainty of providing solid evidence that supports their answers [8]. Clinical evidence predominantly relies on statistical and numerical data. Thus, it is imperative to examine whether LLMs can deliver this type of reasoning. It has been shown that LLMs are more capable when given semantic questions rather than numerical questions, though in a relatively small sample size (smaller than 200 Q and As) [26]. As far as we know, we were the first to show this trend in the medical field while using a much larger scale (Table 1). Furthermore, since both semantic and numeric questions in the EBMQA may address the same entities but from different perspectives, our study questioned whether LLMs can support their semantic answers with statistical data.

In addition, as LLMs are gaining more popularity as decision-support tools [8,9], understanding which types of questions will yield more precise answers, as demonstrated in our semantic and numeric analysis, could benefit not only the medical community but also the general use of LLMs.

A recent benchmark analysis, focused on nephrology Q and As, only found that GPT-4 outperformed Claude 2 [27]. Although our intermodel examination did not include a direct nephrology compression due to a different classification method, it reveals that generally Claude 3 outperformed GPT-4, and specifically in a variety of medical disciplines such as neoplastic disorders, nervous system, and more. Our results raise the need to constantly benchmark new LLMs as they continuously improve.

Regarding internal model variations, the differences in accuracy between the highest and lowest performing medical disciplines, 8% for Claude 3 and 6% for GPT-4, support previous benchmarks that found LLM performance can vary across different medical disciplines [27,28].

Moreover, this comprehensive benchmark widens the medical scope and further supports both intra- and inter-model differences by exploring medical subjects: Claude 3 favors “Imaging and Procedures” and struggles with “Disorders” (64%, 463/719 vs 60%, 3181/5296; P=.03, respectively), while GPT-4 excels in “Imaging and Procedures” but struggles with “Lab tests” (60%, 1017/1683 vs 53%, 1008/1886; P<.0001, respectively). Thus, our study underscores two vital, yet distinct, aspects in the integration of LLMs into daily medical practice: first, the specific areas of medical expertise where each LLM excels, and second, comparing which LLM is superior within each area of medical expertise.

As suggested by previous studies, LLMs are highly sensitive to question wording, structure, and subject matter. Consequently, direct comparisons across different benchmarks, which rely on distinct datasets, may yield varying scores that do not necessarily reflect a genuine knowledge gap but rather other confounding factors such as those mentioned at the start of this section [29,30]. For example, Katz et al [28] reported that GPT-4’s accuracy rates ranged from 17.42% (n=21) to 74.7% (n=90) across various medical disciplines. In contrast, our study found GPT-4’s accuracy rates ranged more narrowly, from 53.5% (n=704) to 60.35% (n=1076). This discrepancy could be partially attributed to the differing medical disciplines emphasized in each study, as well as variations in question structure. While Katz et al [28] used five different exams with potentially diverse question formats, all questions in our study were generated using the same templates, resulting in a relatively narrower accuracy range. Furthermore, Liu et al [31] benchmarked GPT-4 and Claude 3 using the Japanese National Medical Examination and reported accuracy rates of 80.0% (n=720) and 83.6% (n=752), respectively. Although our study observed a similar trend, the accuracy rates differed, with 56.74% (n=12,038) for GPT-4 and 61.29% (n=8583) for Claude 3. A key distinction between the 2 benchmark studies lies in the question structure: Liu et al’s dataset included questions with multiple correct answers, whereas our numerical questions had only a single correct answer. These examples highlight both the importance of treating direct comparisons among LLM benchmark studies with caution and the value of developing multiple high-quality benchmarks on unique, well-designed datasets.

As the debate over whether models surpass humans persists [27,28], the outcomes of our validation tests suggest that humans still excel in certain medical tasks. Therefore, we support further evaluations of LLMs before using them in medical settings.

Furthermore, the insignificant correlation between accuracy and AR contradicts the theory that a model’s confidence in its response reflects its subject expertise [32]. Thus, abstaining from providing an answer failed to explain the intramodel variance results, specifically across medical disciplines. Notably, recent research has shown that LLMs exhibit varied abstention abilities, which is consistent with our finding and may be influenced by model-specific characteristics, context nature, and question type [33]. For instance, some LLMs find it challenging to abstain from Boolean questions with standard prompts. Intriguingly, modifying the context by introducing irrelevant information can occasionally enhance abstention performance and, thereby, improve overall task accuracy [33].

This evidence, along with our findings, raises concerns that without prior knowledge of both the medical field and the model, the trustworthiness of LLMs is questionable.

In terms of prompt engineering, our sensitivity analysis showed relatively small SDs in prompt accuracy, which supported our prompt stability. In addition, although insignificant, the IDK prompt yielded higher accuracy and was therefore used. Moreover, changing the order of the distractors did not significantly affect the LLM’s performance.

Our benchmark has several limitations. First, although medically tuned LLMs have shown promising results [34], they are not publicly available and hence were not included in this study. We highly recommend conducting a similar benchmark that includes these LLMs. Second, we did not use additional context for the prompt or use external methods such as retrieval-augmented generation, which could potentially improve the results. We chose not to use these methods because we believe that, currently, physicians are asking LLMs straightforward questions. In addition, some of these external methods are not widely accessible to end users and are far more complex than the typical daily use of LLMs that we aimed to replicate. Given that these methods might influence the results, we strongly recommend conducting research that focuses on retrieval-augmented generation or providing extra context in the prompt. Third, the study was designed so that the models and human participants would only choose one suggested answer, without providing additional information or feedback. Therefore, we support further studies to examine these responses by the models, while considering feedback from human physicians. Fourth, this study did not include a subanalysis regarding progressive patterns such as the abstention behavior. Therefore, we recommend further research on the subject. Another limitation is the known potential biases when using LLMs, such as the training on an enormous amount of data, which may include bias and inaccuracies itself, or may also harm the contextual understanding of medical cases and result in poorer answers due to undertraining on less common medical disciplines [35].

On the EBMQA dataset, which resembles physicians’ problem-solving approach, LLMs were better at solving semantic than numeric questions. Despite Claude 3 surpassing GPT-4, both LLMs exhibited inter- and intramodel gaps in medical knowledge. In addition, human participants outperformed both LLMs on numeric questions. These results suggest that LLMs’ responses, especially numeric ones, should be considered cautiously in clinical settings.