The network meta-analysis (NMA) was based on the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) reporting guidelines. The PRISMA checklist is shown in Multimedia Appendix 1. The Bayesian approach permits the indirect comparison of performance between a range of LLMs that were not explicitly articulated throughout the experiment. The study protocol was defined and registered in the PROSPERO database prior to the commencement of the study.

A computer search of the PubMed, Embase, Web of Science, and Scopus databases was conducted to identify relevant studies on the accuracy of different LLMs when answering questions in the medical field. The last search was updated to October 14, 2024, to identify studies published since the first search, with no restrictions on the type of study. When the results of a study were reported in multiple publications, we included the study with the richest and most recent findings. We also searched the list of literature on LLMs in medicine-related systematic reviews and manually searched the references included in the reviews for additional access to relevant literature. The search subject terms were “LLM,” “generative AI,” “open AI,” “Large language model,” “ChatGPT-3.5,” “ChatGPT-4,” “Google Bard,” and “Bing,” without any language restriction. The complete search strategies for all databases are shown in Multimedia Appendix 2.

A combination of EndNote X9 deduplication and manual deduplication was used to screen the literature in accordance with the developed inclusion criteria. The results of the literature searches conducted in different databases were then combined to create a new information database, which could be downloaded in full text. Independent review and assessment of the titles, abstracts, and full texts of the relevant literature were undertaken by 4 authors (LW, JL, BZ, and SH). The review encompassed studies using disparate LLMs systems to respond to medical queries. Letters, conference abstracts, editorials, reviews, and expert opinions for which no information was available were excluded from the review. In addition, the following studies were excluded: those that evaluated the performance of only 1 LLM; those that assessed the performance of 2 or more LLMs without specifying the LLM versions used (eg, the article only mentioned evaluating ChatGPT without mentioning ChatGPT-3.5, ChatGPT-4, or other versions), with the updated versions and timelines of various LLMs so far shown in Multimedia Appendix 3; those that assessed the performance of 2 or more LLMs but did not provide data isolating their accuracy when answering different types of questions; and the questions included in the study contained images. In addition, to reduce bias, we excluded research on accessing LLMs through an application programming interface (API).

The primary outcomes were the accuracy of LLMs when answering medical questions. These included objective questions, open-ended questions, top 1 diagnosis, top 3 diagnosis, top 5 diagnosis, and triage and classification accuracy. Objective questions are exam questions with a clear, quantifiable answer that is usually predetermined, unique, or with a limited number of options. Open-ended questions are a type of question that does not have a fixed answer nor standardized answer. Diagnosis and triage and classification are open-ended questions, but most diagnostic questions end with “What is the most probable diagnosis?” whereas triage and classification questions end with “How would you classify this disease?” Corresponding examples are shown in Multimedia Appendix 4.

Accuracy for objective questions was calculated as the number of correctly answered questions divided by the total number of questions. For diagnosis and classification, accuracy was defined as the number of cases correctly diagnosed or triaged divided by the total number of cases. Specifically for open-ended questions, accuracy was determined based on the number of questions rated “good” or “accurate” on the accuracy scale divided by the total number of questions.

The 4 researchers jointly extracted and verified the following data: (1) basic information about the included studies, such as study title and first author; (2) baseline characteristics and interventions of the study population; (3) key elements evaluated for risk of bias; and (4) outcome indicators and relevant outcome measure data. Our study involved extracting raw data from each study. In cases of disagreement, these were resolved through discussion and consultation with a third party.

Because they were cross-sectional studies, the quality of the included studies was evaluated using the Newcastle-Ottawa Scale [19]. The quality assessment was conducted by 3 independent researchers (LW, JL, and BZ), with a fourth researcher (SH) resolving any disagreements. A low overall risk of bias was determined when the Newcastle-Ottawa Scale score ranged from 7 to 9, moderate risk was determined when the score was between 4 and 6, and high risk was determined when the score was 0 to 3.

Statistical analyses were performed using Stata 18.0 and R (version 4.3.1), with the odds ratio (OR) as the analytical statistic. Accuracy was assessed using 95% CIs and the credible interval. NMA analyses were performed on different types of LLMs.

The confidence of the NMA results estimates was assessed according to the Confidence in Network Meta-Analysis (CINEMA) methodology, which is broadly based on the Grading of Recommendations Assessment, Development, and Evaluation (GRADE). An NMA was conducted within a Bayesian framework using Markov chain Monte Carlo methods and was computed using the BUGSnet and GeMTC packages in R (V.4.3.1) software. A network graph was constructed for each LLM included in the experiment in order to facilitate a comparison of the performance of multiple LLMs. The consistency between direct and indirect evidence was evaluated using a node-splitting method when there was a closed loop. If the P value between the direct, indirect, and network comparisons of the 2 interventions was >.05, we concluded that there was no statistical difference and consistency was good. The convergence of the network models derived from the Markov chain Monte Carlo simulations was assessed using trace and density plots. We used noninformative priors for all parameters and assumed common heterogeneity. Furthermore, for all LLMs, we determined the ranking probabilities, which were articulated as the surface under the cumulative ranking curve (SUCRA). Higher SUCRA values suggest superior accuracy in model ranking.

A bibliographic search yielded 59,075 citations, of which 21,156 studies were identified as potential conditions based on abstract screening and retrieved for full text evaluation. Manual reading of the titles and abstracts of the remaining literature excluded 20,814 papers whose topics and interventions did not match the inclusion criteria for this study. Further reading of the full texts excluded the following: 174 articles that could not be separated nor extracted from the ending; 147 articles in which we were unable to separate outcome data, unable to extract outcome data, or detected issues related to images; 12 articles with unclear versions of the LLMs; and 8 articles that used an API to access LLMs. In addition, the full text of 7 articles was not available, resulting in the final inclusion of 168 articles from the literature. The literature screening process is shown in Figure 1.

To assess the accuracy of different LLMs when answering medical questions, a total of 168 studies underwent a screening process to determine their suitability for inclusion. A total of 35,896 questions and 3063 clinical cases were included in the study. The basic information of the 168 studies is presented in Multimedia Appendix 5.

In the quality assessment, 40 (40/168, 23.8%) studies were assessed as having a low overall risk of bias, while 128 (128/168, 76.2%) had a moderate overall risk of bias. No studies were identified as having a high overall risk of bias. The detailed quality assessment results for each study can be found in Multimedia Appendix 6.

This study presents the most comprehensive meta-analysis to date on the accuracy of various LLMs when responding to medical queries, encompassing objective questions, open-ended questions, top 1 diagnosis, top 3 diagnosis, top 5 diagnosis, and triage and classification. Variations in accuracy among different LLMs were observed. ChatGPT-4o demonstrated the highest accuracy when answering objective questions, while ChatGPT-4 excelled at open-ended questions. The superior performance of people at the top 1 diagnosis and top 3 diagnosis suggests that human expertise is generally more dependable than LLMs in complex medical scenarios, while Claude 3 Opus seems to perform the best in the top 5 diagnosis. In terms of triage and classification, Gemini appeared to be more reliable.

In addition, we stratified LLMs according to the medical field in which the objective questions were located and explored their accuracy in 6 fields: ophthalmology, orthopedics, urology, dentistry, oncology, and radiology. We found that Aeyeconsult performed the best in ophthalmology, Bard performed the best in orthopedics, Bing chat performed the best in urology, ChatGPT-4 performed the best in both dentistry and oncology, and ChatGPT-4o had the highest accuracy in radiology.

At present, language models based on transformer architecture, whether pretrained or fine-tuned using biomedical corpora, have been proven effective in a series of natural language processing benchmarks in the biomedical field [183]. We attempted to analyze the reasons for the performance differences when different LLMs answer questions. Parameter size is an important factor affecting the accuracy of LLMs when answering questions. Research has found that, when the parameter size of the PaLM model is expanded from 8B to 40B, the accuracy of answering medical questions is doubled [184]. However, the practicality of a model depends not only on its number of parameters but also on many factors such as its training data and architecture, fine-tuning protocols, and overall architecture [185]. Taking GPT-4 as an example, it achieved a higher performance than its predecessor by adopting more advanced training data and architecture. The timeliness and accuracy of training data are also crucial for model performance. Today, models can not only rely on a limited set of pretraining data but also obtain the latest knowledge from the internet in real time. For example, Bing AI and Google Bard already have the ability to obtain real-time updates, and ChatGPT has also begun to follow suit by accepting plugins to expand its capabilities [185,186].

In addition, we found that some models fine-tuned on the backend LLM can achieve higher accuracy and less energy consumption in specific fields. For example, in the field of ophthalmology, Aeyeconsult integrates many ophthalmic data sets based on GPT-4 for training and generation [24]. This targeted training can significantly improve its performance in ophthalmic clinical tasks. Other possible data sources include clinical texts and accurate medical information, such as guidelines and peer-reviewed literature. In fact, there are already some models built or fine-tuned based on clinical text, such as SkinGPT-4 and ChatDoctor, which perform better overall than various general LLMs at biomedical natural language processing tasks [187,188].

Progress on various grand prognostic models has been very rapid, with a newer, more arithmetically powerful version being released every few months. However, our results show that the newer versions do not necessarily outperform the older ones in terms of performance when measured as accuracy, possibly because the newer versions incorporate fewer studies, which may have biased the results somewhat. In addition, updated versions such as ChatGPT-4V provided multimodal models (eg, that can evaluate image problems), and these models may have a greater advantage for image evaluation, for example.

Studies indicate LLMs outperform humans at exams like medical licensing, orthopedics, and pediatrics globally, highlighting LLMs’ potential as a study aid. For the top 1 diagnosis and top 3 diagnosis, human accuracy is higher than that of LLMs. Despite the fact that Claude 3 Opus outperformed humans in the top 5 diagnostic results, due to the high level of accuracy required in the medical field and the multifaceted information and complex decision-making involved in medical diagnosis, we still recommend that LLMs should only be used as an auxiliary tool to assist doctors with more efficient data analysis and preliminary diagnostic recommendations.

Several meta-analyses have been conducted to assess the accuracy of LLMs in health care [15,189,190]. However, it is very unfortunate that the LLMs included in these studies included ChatGPT only and that some of the studies simply evaluated its performance on exams. Some studies did not differentiate between the types of questions answered by ChatGPT, which led to a significant amount of heterogeneity between the studies, resulting in biased results.

We acknowledge certain limitations in our study. First, for the top 3 diagnosis, top 5 diagnosis, and triage and classification, this may bias the results due to the number of included studies as well as the sample size, so caution is needed when interpreting these results. Although we minimized the heterogeneity of the research as much as possible, we cannot deny that the inclusion of different fields of study and the complexity of LLMs (such as different instructions and questioning dates) can affect the results of the study and generate heterogeneity. Therefore, caution should be exercised when interpreting the results. In addition, we did not assess the accuracy of multimodal grand prognostic models when solving medical image–related problems; with the development of artificial intelligence, more multimodal models are being developed, and in the future, these models will become indispensable in the exploration of image-based problems in the medical field.

Existing studies suggest that ChatGPT-4o has an advantage for answering objective questions. For open-ended questions, ChatGPT-4 may be more credible. Humans are more accurate in the top 1 diagnosis and top 3 diagnosis of clinical cases. Claude 3 Opus performs better in the top 5 diagnosis, while for classification accuracy, Gemini is more advantageous. Although some LLMs excel at addressing medical queries, caution is advised due to the critical need for precision and rigor in medicine. Future high-quality studies and trials are necessary to gather more scientific evidence.