This scoping review was conducted according to the Joanna Briggs Institute Manual for Evidence Synthesis and adheres to the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) [14].

Web of Science, PubMed, Embase, IEEE Xplore, and ACM Digital Library databases were searched as the source of relevant literature from 2022 to January 2025. The first search was conducted on September 27, 2024, and the final search on January 12, 2025. Multimedia Appendix 1 provides the detailed search strategies. Raw data were first imported into the Rayyan platform. Subsequently, 4 authors (HS, YS, AZ, and YY) independently screened for potentially relevant articles in a blind mode with the following procedure: duplicates were first removed, followed by a screening of browsing titles and abstracts. The remaining articles were screened for the second time to evaluate the full text of all eligible articles. At any point in the process, conflicting articles were resolved by RL. The eligibility criteria are listed in subsequent section.

The inclusion criteria were as follows: (1) articles that focused on reporting the outcome of LLMs in medical diagnosis–related field and (2) articles that described the application of LLMs for medical diagnosis. LLMs here are defined as deep learning models with more than one million parameters, trained on unlabeled text data

To obtain only necessary data, we excluded articles that were conference abstracts or abstracts only and (2) comments and retracted articles.

After the selection of included articles, the data from the included articles from Web of Science were first exported in tab-delimited format and then imported into Microsoft Excel 2019 for preliminary collation. Non–Web of Science articles were manually entered into Microsoft Excel 2019 by 2 independent researchers (AZ and RL) to ensure accuracy. The combined dataset was converted to tab-delimited format for compatibility with a bibliometric tool. VoSviewer (Leiden University) was used to map research trends and frontiers in LLM applications for medical diagnosis. This analysis identified 3 dominant research clusters reflecting key thematic foci, which subsequently informed the design of data extraction categories. The bibliometric analysis objectively identified field-level hot spots and trends, while the scoping review contextualized these patterns through qualitative appraisal of study designs and clinical validations. This dual approach ensured both macroscopic trend detection (via VOSviewer keyword clustering) and microscopic evidence evaluation (via structured data extraction).

The bibliometric clusters guided the development of an extraction template to systematically capture bibliometric attributes (eg, authors, publication years, and countries), thematic focus (eg, specialty task and LLM type), and clinical validation metrics (eg, primary end point, comparator, and primary result). Two authors (HS and YS) independently extracted data using this template, resolving discrepancies through consensus discussions. A pilot extraction on 10% of articles preceded full-text review to refine category definitions.

Figure 1 displays the study selection process. A total of 17,757 records were identified from Web of Science, PubMed, Embase, IEEE Xplore, and ACM Digital Library databases. After removing 2445 duplicates, we screened titles and abstracts, and 6708 records were assessed for inclusion, resulting in 95 articles in the final review. The overall characteristic of included articles is provided in Multimedia Appendix 2 [15-108]. Figure 2A illustrates the distribution of articles by year, which indicates that this research area is still emerging, with a significant increase in publications occurring in 2024. This trend suggests that the field is likely to continue experiencing positive growth over the coming years. Figure 2B illustrates that the United States and China are the most productive countries in the research of LLM in diagnosis. In addition, the United States has the highest number of interconnected targets in country cooperation. It is suggested for other countries’ researchers to strengthen international cooperation and communication for this research topic.

We used VOSviewer software for bibliometric labeling of included studies, and a total of 65 keywords and 3 clusters were obtained (Figure 3). Cluster 1, “The assessment of LLMs in medical diagnosis,” is noted in green, with keywords including “accuracy” and “diagnostic accuracy.” This cluster focuses on the accuracy and reliability of LLMs in medical diagnosis and how they support clinical decision-making and triage. Cluster 2, “The application of LLMs in medical diagnosis,” is represented by red, with keywords including “orthopedics,” “cardiology,” and “ophthalmology,” among others. This cluster illustrates the application of LLMs in diagnosis across various medical specialties, including orthopedics, cardiology, and ophthalmology. Cluster 3, “Impacts of using LLMs in medical diagnosis,” is indicated in yellow, with keywords including “health,” and “outcomes,” among others. This cluster focuses on the practical outcomes and impacts of using LLMs in medical diagnosis, including their impact on care, health outcomes, and telemedicine. These clusters are in line with other reported analysis and reviews, suggesting a high reliability [109,110].

We subsequently labeled the articles into the 3 clusters (overlapping articles remained in all mother clusters) for further analysis. Results are provided in Multimedia Appendix 2.

As shown in Table 1, among the 95 included articles, GPT-4 and its variant models were studied in 70 (74%) studies, followed by GPT-3.5 and its variant models in 34 (36%) studies, specific developed GPTs in 9 (9%) studies, Claude and its variant models in 9 (9%) studies, Gemini and its variant models in 8 (8%) studies, and Llama and its variant models in 8 (8%) of the included studies. This suggests that ChatGPT is currently the most popular LLM used in medical diagnosis.

It is also worth noting that specifically developed GPT versions have also surged in numbers in recent years, indicating a translation from general to specific LLMs. Among the specifically developed GPT versions, we extracted 3 major paths for developing a specific GPT version, as described in Table 2 and Textbox 1.

The applications of LLMs in medical diagnosis varied and showed promise across several medical specialties. The frequency of LLM applications in different medical domains, as depicted in Table 5, indicates a high interest and potential in using these models for diagnostic purposes.

Among all the medical domains, research has focused on radiology, which accounts for 16 (17%) studies. In radiology, the application of LLMs is mainly focused on image recognition, interpretation, and diagnosis. LLMs can effectively identify pathological areas and improve diagnostic accuracy [148]. At the same time, LLMs can also conduct risk prediction and formulate personalized treatment plans based on patients’ medical imaging data [149]. In addition, through natural language generation technology, LLMs can also provide patients with understandable diagnostic reports and health guidance, which would aid the diagnostic process [150].

It is also worth noting that the application of LLMs on psychiatry diagnosis, including depression and attention-deficit/hyperactivity disorder are very promising [151-153]. Similarly, diseases of the nervous system also attract considerable attention, with studies covering disorders from predicting seizure to determining Alzheimer disease [154-156]. This might be because decision-making in psychiatry and neurology is particularly unique, as they have fewer objective tools that can be used to either confirm or refute a diagnosis [156]. Therefore, the emergence of LLMs can be an objective assessment tool that can help experts in decision-making process for diagnoses [121].

Clinical trials related to LLMs were searched in ClinicalTrials.gov (March 17, 2025) as indicators for LLMs’ real-world integration (Tables 6-8). After removing unrelated trials, 23 clinical trials were selected (Multimedia Appendix 4 provides detailed information on trials). These trials spanned across 11 medical specialties, with oncology (6/23, 26%) and ophthalmology (5/23, 22%) being the most extensively studied. All trials were funded by public sponsors and conducted in publicly available hospitals. None of the trials had reported their results by the date of search. Most trials (15/23, 65%) used LLMs as support in physicians whole diagnostic process, while some (6/23, 26%) investigated LLMs’ potential in direct diagnosis of diseases conducted by a physician. We observed that LLMs used for direct diagnosis were centered in emergency medicine and radiology. These areas showed shortages in physicians, where LLMs served as aid to reduce the workload of physicians [157,158]. For example, LLMs’ quick reasoning and responding makes them an ideal choice for grading emergency patients, and their enhanced vision understanding capabilities are valuable for generating radiology reports. It was found that ChatGPT and its variants still dominated the use in clinical trials (10/23, 43%), which is consistent with our previous findings. However, we observed a rise in using specifically develop LLMs in clinical settings compared with our previous findings (6/23, 26% vs. 9/95, 10%); this might be because the clinical use of LLMs was specialized in particular subject. Therefore, applying fine-tuning or prompt engineering method could enhance the usability of LLMs in expertized clinical scenario. In addition, it should be noted that certain trials did not report the LLMs used in the study or reported them without proper disclosure of their type (eg, only mentioned ChatGPT without its model types). Privacy concerns have driven innovations, such as locally deployed LLMs (ie, the ongoing trial registered as NCT06865534). These models minimize data transmission risks by processing information on-site, aligning with regulations like General Data Protection Regulation (GDPR). However, only this trial explicitly mentioned privacy safeguards, indicating a need for standardized reporting on data security measures in LLM research.

The field of LLMs in diagnosis has witnessed tremendous development in recent years, which is evident from our findings on study distribution across years. The results of our study contribute to a growing body of literature on the application of LLMs in health care. Previous scoping reviews have demonstrated that considerable research in the field of LLMs in health care is related to medical diagnosis [10]. In addition, there are existing literatures examining the current application of LLMs in diagnosis [159,160]. However, there is still a noticeable absence of a comprehensive analysis of LLMs in diagnosis to identify the gap between research and mapping the current research landscape of LLMs in medical diagnosis.

This is a pioneering study to quantitatively and comprehensively chart the integration of LLMs into the medical diagnosis domain. In this scoping review, we noticed a surge in publications regarding LLMs in diagnosis, signifying growth potential in this research field. This surge in research activity underscores the growing interest and potential impact of LLMs in transforming diagnostic processes in health care.

It is also worth noting that we performed a bibliometric analysis before our data extraction process. This was partly inspired by the bibliometric-systematic literature review approach proposed by Marzi et al [161]. In short, instead of gathering data directly from the 95 included articles, we performed a bibliometric analysis to cluster their keywords as indicators for determining research questions and data to extract. By using this technique, we were able to extract our needed information from a vast amount of data to support our findings. In addition, by performing a nested bibliometric analysis, this study was able to map the current landscape of LLM diagnostic performance evaluation, which provides valuable insight and is compatible with existing literature recommending a framework for future LLM performance evaluation [162].

Our review reveals that LLMs, particularly ChatGPT and its variants, show substantial promise in enhancing diagnostic accuracy and facilitating clinical decision-making. Traditional clinical decision support systems using decision trees increase users’ accuracy in diagnosing diseases [163] but struggle with complex presentations requiring probabilistic reasoning [164]. In contrast, LLMs demonstrate a superior diagnostic accuracy rate with a clearer indication of reasoning. However, the hallucination problem in LLMs creates regulatory challenges for high-stakes applications [165]. Nevertheless, the ability of LLMs to process vast amounts of textual and visual data enables them to classify diseases effectively, as demonstrated in various medical specialties such as radiology, psychiatry, neurology, and dermatology. For instance, GPT-4 has been successfully applied to detect and classify diseases from EHRs and medical images. Currently, the integration of LLMs with other technologies, such as computer-aided diagnosis networks and attention mechanisms, creates hybrid systems that combine the strengths of both approaches [130], which further enhances their diagnostic capabilities and suggests a promising direction for LLM development.

The rapid advancement in the accuracy of LLMs in medical QA tasks is another notable achievement. Models like MedPaLM, GPT-4, and MedPaLM2 have shown significant improvements in their ability to provide accurate and relevant answers to medical queries. This progress is partly attributed to the development of medical task–specific LLMs, which are fine-tuned on specialized datasets to enhance their domain knowledge and reasoning abilities. For example, ChatDoctor and AcupunctureGPT, which incorporate real-time information retrieval and traditional Chinese medicine principles, respectively, demonstrate the potential of LLMs to adapt to diverse medical contexts and improve diagnostic outcomes.

Despite promising advancements, several critical challenges remain. One of the most significant concerns is the potential for bias in LLMs, which can lead to inequitable and unethical health care outcomes. Our review highlights that LLMs are susceptible to reflecting and amplifying biases present in their training data, resulting in biased diagnostic recommendations and disparities in patient care. For instance, studies have shown that LLMs can exhibit race-based biases in medical diagnosis and patient information processing. Specifically, LLMs have been found to generate biased patient backgrounds and associate diseases with specific racial or ethnic groups. For example, GPT-3.5-turbo has been shown to attribute unwarranted details to patients based on their race or ethnicity, such as associating Black male patients with a safari trip in South Africa, and varying its diagnoses for different racial and ethnic groups even under identical conditions (eg, diagnosing HIV in Black patients, tuberculosis in Asian patients, and cysts in White patients) [145]. In addition, LLMs may project higher costs and longer hospitalizations for certain racial groups. For example, GPT-3.5-turbo predicts higher costs and longer hospital stays for White patients compared to other racial and ethnic groups, potentially reflecting real-world health care disparities. Furthermore, LLMs have been found to show overly optimistic outcomes in challenging medical scenarios with higher survival rates for some groups compared to others, which may negatively impact the quality of diagnoses and treatment decisions [144]. It should also be noted that apart from the race and gender bias that were directly reported by researchers studying medical diagnosis, LLMs had also exhibited bias in patient care regarding age, social status, and income level. These are also possible hidden bias that might affect the decision-making of LLMs in medical diagnosis.

Addressing these biases is crucial to ensure that LLMs provide fair and accurate diagnostic support to all patients. Currently, the most widely adopted method for bias mitigation is through prompt tuning. By carefully designing and adjusting the prompts used to query the LLMs, it is possible to guide the models toward more equitable and accurate responses. For example, researchers have found that using prompts that explicitly emphasize fairness and inclusivity can reduce the likelihood of biased outputs [166]. In addition, incorporating counternarratives or examples that challenge stereotypes within the prompts can help counteract the biases that may be present in the training data [167]. However, such methods only provide a shield to prevent the occurrence of bias in diagnosis without changing its core. Reinforcement learning from human feedback is another strategy that allows humans to grade the model’s responses, which helps correct some model outputs, particularly on sensitive questions with known online misinformation. However, such a method costs additional computing and could result in a more ambiguous answer in diagnosis. For example, the racial bias in cost prediction for GPT-4 has been reduced from 18% in GPT-3.5 to 5%, but the “uncertainty rate” increased from 16.25% to 29.46% [145]. Therefore, future research in finding a suitable way fully eradicate biased medical diagnosis, and constant monitoring of LLMs bias performance are important for the application of LLM in medical diagnosis.

In addition, the complexity of real-world clinical scenarios presents a challenge for LLMs. From a technical perspective, the first hurdle for LLMs deployment in real-world clinical settings is the ambiguity and uncertainty in unstructured inputs [15]. Unlike structured QA tasks, patient concerns (eg, “chest pain”) may correspond to dozens of potential diseases. LLMs must generate reasonable differential diagnoses despite lacking definitive information. Additional hurdles include the need for dynamic interaction and real-time feedback. LLMs in clinical consultations require interactive information gathering in follow-up history and test results, among others. Current LLMs’ limited context windows (eg, GPT-4’s 32,000 tokens) may lead to critical information loss. In addition, dynamically synthesized laboratory results, imaging reports, and other multimodal data are required for an accurate and reasonable diagnosis, but cross-modal alignment techniques remain underdeveloped in LLMs. Therefore, ensuring that LLMs can effectively navigate these complexities without compromising diagnostic accuracy is a critical area for future research.

Beyond technical challenges, the clinical application of LLMs faces multifaceted nontechnological barriers, including fragmented regulatory frameworks, physician skepticism, data privacy risks, ethical dilemmas, and regulatory inconsistencies across regions, such as the United States. The requirement of software as a medical device for consistent and reproducible results for medical software contradicts LLMs’ nature as products generating differentiated content for each answer. In addition, the United States’ requirement for predefined change control plans for adaptive algorithms, the European Union’s stringent transparency and data traceability mandates under the AI Act and the GDPR, and China’s life cycle quality control protocols create complex compliance landscapes that delay global deployment [168,169]. Clinician trust is further eroded by the black box nature of LLMs, concerns over data representativeness (eg, biases arising from existing studies), and ambiguous liability frameworks for errors [170]. Data privacy remains a critical hurdle, as LLMs rely on sensitive patient information while navigating strict anonymization rules (eg, the GDPR) and cross-border data flow restrictions. Current LLMs in medical diagnosis are still mainly commercially available models, with only a few reports on local deployment of LLMs, which raises the concern of patient data leakage. Ethical risks, such as algorithmic biases in gender or race and inadequate patient consent processes, compound public distrust, particularly among vulnerable populations [19]. These challenges underscore the need for international regulatory harmonization, enhanced algorithmic transparency through adversarial testing, and collaborative ethical governance to balance innovation with patient safety and trust in real-world clinical settings.

The scoping review process, while comprehensive, has inherent limitations. First, our reliance on selected databases might have excluded pertinent publications from nonindexed sources, for example, arXiv. Therefore, the source for this scoping review is limited. Despite that, we have tried to include articles from other sources by screening the citation of included studies; however, certain valuable gray literature or ongoing research may be omitted. In addition, this review lacked longitudinal studies evaluating the long-term clinical impact of LLMs, such as sustained diagnostic accuracy, clinician reliance, and patient outcomes over extended periods. This gap restricts conclusions about the durability of LLM performance and its integration into dynamic health care workflows. Moreover, because research into LLMs in medical diagnosis is only beginning, the rapid evolution of LLM technology introduces a temporal limitation: studies published after January 2025 or advancements in multimodal architectures may already outpace findings in this review. In addition, due to the expected heterogeneity in tasks and end points, we did not conduct formal meta-analyses or other statistical analysis. Instead, we presented simple descriptive statistics to provide an overview of the features of the LLMs’ landscape in medical diagnosis. Future updates with statistical analysis are critical to maintaining relevance in this fast-moving field.

In conclusion, this scoping review highlights the significant potential of LLMs to revolutionize medical diagnostics, while also emphasizing the critical need to address biases, privacy concerns, and the complexities of real-world clinical scenarios. By focusing on these areas, future research can pave the way for the successful integration of LLMs into health care systems, ultimately improving diagnostic accuracy and patient outcomes.