The Institutional Review Board of Henan Provincial People’s Hospital determined that this in silicon research did not involve direct interaction with real-world human subjects, nor did it require the collection of new human data. Accordingly, an ethics exemption was granted for this study. The case information used was derived from publicly available sources and published materials. At the time of data collection, GPT-4.0 was publicly available by paid subscription through ChatGPT Plus.

The clinical cases for our study were sourced from Chinese Standardized Resident Training–Ophthalmology (2nd Edition), which is an official resource conforming to SRT content and standards, as well as the theoretical assessment guidelines of the National Health Commission of China. Unlike traditional undergraduate textbooks, SRT materials are specifically designed to focus on problem-based learning and case-based learning. They feature a variety of typical real-world cases from various ophthalmic subspecialties, making them particularly suitable for interaction with ChatGPT’s chatbot interface, which is designed to handle complex, real-life queries.

In this study, we gathered 121 cases from 12 ophthalmic subspecialties, creating 2 profiles per case: patient with history (Hx) and patient with history and examination (Hx+Ex) [12]. The “history” portion comprised gender, age, chief complaints, and medical history. When necessary, past medical, familial, and systemic disease details were also added. The “examination” portion covered general ophthalmic assessments such as visual acuity and intraocular pressure, alongside diagnostics such as slit lamp biomicroscopy, a range of ophthalmic imaging (eg, fundus photography, orbital computed tomography [OCT], and fluorescein angiography), and specialized imaging for eye tumors and traumas (OCT and magnetic resonance imaging). The inclusion criteria mandated comprehensive historical and ophthalmology-related chief complaint details, documented examination results, and official case analyses with accurate subspecialty classifications and unique diagnoses.

After excluding 17 cases, a total of 104 cases were retained for further analysis. The reasons for excluding the 17 cases included (1) the lack of medical history or the presence of final diagnostic–like terms in the medical history, which may bias the assessment (6 cases); (2) the lack of textual descriptions of examination results (6 cases); (3) chief complaints not related to ophthalmic symptoms or cases referred from other departments (3 cases); and (4) unclear or non–ophthalmology-related diagnoses (2 cases). The mean age was 36.8 (SD 21.7) years, with male patients comprising 55.8% (58/104) of the cases. Notably, each case had a predominantly unique final diagnosis. The 3 most prevalent diagnoses, classified into subspecialties, were eyelid, lacrimal, and orbital diseases; retinal diseases; and strabismus and amblyopia (Table 1). Given that the original 104 profiles were ordered based on the ophthalmic subspecialty, a new case numbering system was established by randomly assigning descending numerical values between 0 and 1 and arranging them accordingly.

A total of 208 clinical profiles (104 Hx and 104 Hx+Ex profiles) in Chinese were tested from March 5 to 18, 2024. The tested AI chatbots included ChatGPT versions 3.5 and 4.0 (GPT-3.5 and GPT-4.0, respectively), and 3 ophthalmology residents were also tested. Initially, ChatGPT was assigned a system role to emulate a professional ophthalmologist. Each clinical case scenario was entered in Chinese independently, followed by 2 questions: “Q1. Which ophthalmic subspecialty should be given priority for registration?” and “Q2: List the top three possible diagnoses and provide a detailed rationale for each.” Q1 aimed to elicit the chatbot’s triage recommendations for subspecialty registration, while Q2 focused on extracting the chatbot’s leading and differential diagnosis proposals. For the Hx profiles, both questions were asked, and the AI was informed of the available subspecialties for reference in the prompt. For the Hx+Ex profiles, given that the examination primarily serves for diagnosis, only Q2 was posed. Response history was reset prior to each new query (Figure 1). The detailed prompts and engineering techniques are shown in Multimedia Appendix 1. Based on the structure provided by these prompts, we found that ChatGPT’s responses generally adhered to the fixed template specified in our prompts. Consequently, we opted to input each case continuously in a single chat session. However, for the Hx and Hx+Ex profiles, as well as for tests conducted with both GPT-3.5 and GPT-4.0, each series of tests was initiated in a new chat session.

For the residents’ test, 2 sets of questionnaires were created following the Hx + Q1 and Hx+Ex + Q2 format. The residents’ evaluations were independent, which were ensured by implementing a blinded assessment process where the residents did not have information about the performance or responses of the AI systems or each other.

The study focused on 3 outcomes: accuracy of recommendation for ophthalmic subspecialty registration, accuracy of the top-ranked diagnosis, and accuracy of the diagnosis within the top 3 suggestions (do-not-miss diagnosis). As residents rarely provided 3 possible diagnoses like ChatGPT, they were not evaluated on the accuracy of the do-not-miss diagnosis. The gold standard for judgment was the official diagnosis from Chinese Standardized Resident Training–Ophthalmology (2nd Edition).

ChatGPT and residents were evaluated based on the same outcome criteria. For ophthalmic subspecialty registration, overlaps exist in some subspecialties, such as eye tumors and retinal diseases. Recommendations to either category were considered correct. Precision was crucial for diagnosis suggestions corresponding to Hx+Ex profiles. For instance, if the final diagnosis is sympathetic ophthalmia, responses such as panuveitis or herpetic keratitis were marked as incorrect. Similarly, for acute idiopathic optic neuritis or orbital neurilemmoma, responses of optic neuritis or orbital tumor were also considered incorrect. However, for Hx profiles, an exact diagnosis was not required. All responses in the examples provided were considered correct.

Regarding the residents’ performance, a diagnosis was considered correct only if at least 2 out of 3 residents provided the correct diagnosis. This approach emphasizes the importance of consensus in clinical decision-making and reflects the collaborative nature of medical diagnosis in real-world settings.

Data collection and management were performed using Microsoft Excel software. Statistical analyses were mainly conducted in SPSS (version 26.0.0; IBM Corp). To compare the accuracy of triage and diagnosis across different testing strategies, the Pearson chi-square test or Fisher exact test was applied, depending on the expected frequency counts in the contingency tables. Due to limitations in SPSS for conducting the Fisher exact test on the differences in proportions among 3 groups, we switched to R software (version 4.4.1; R Foundation for Statistical Computing) for this analysis. For post hoc analysis, P values were adjusted using the Bonferroni method in pairwise comparisons. Unless otherwise specified, differences were considered statistically significant at P<.05.

For Hx profiles, GPT-3.5, GPT-4.0, and residents demonstrated moderate accuracy in registering patients to the correct ophthalmic subspecialty, with accuracy of 63.5% (66/104), 77.9% (81/104), and 69.2% (72/104), respectively (P=.07). Subgroup analysis revealed that ocular trauma, retinal diseases, and strabismus and amblyopia ranked among the top 3 in overall registration accuracy. In contrast, registration accuracy was low for glaucoma, neuro-ophthalmology, and lens diseases (Table 2). The detailed registration recommendations for GPT-3.5, GPT-4.0, and residents are showed in Multimedia Appendix 2. An example of GPT-4.0’s response to subspecialty registration is shown in Figure 2.

For Hx+Ex profiles, both GPT-4.0 and residents demonstrated higher diagnostic accuracy compared to GPT-3.5 (62/104, 59.6% vs 41/104, 39.4%; P=.003; and 63/104, 60.6% vs 41/104, 39.4%; P=.001, respectively). Similarly, the accuracy of diagnoses within the top 3 suggestions were also higher (79/104, 76% vs 51/104, 49%; P<.001; and 68/104, 65.4% vs 51/104, 49%; P=.02, respectively). However, there was no statistically significant difference in diagnostic accuracy between GPT-4.0 and residents (79/104, 76% vs 68/104, 65.4%; P=.09). Compared to Hx profiles, GPT-4.0 showed improved diagnostic accuracy for Hx+Ex profiles and diagnoses within the top 3 suggestions (62/104, 59.6% vs 42/104, 40.4%; P=.007; and 79/104, 76% vs 63/104, 60.6%; P=.02, respectively; Table 3). The detailed top-3 predicted diagnoses by GPT-3.5 and GPT-4.0, alongside the composite diagnosis by the 3 residents, are showed in Multimedia Appendix 3. An example of GPT-4.0’s response to diagnosis is shown in Figure 2.

In the subgroup analysis, both GPT-3.5 and GPT-4.0 exhibited generally lower diagnostic accuracy for Hx profiles. However, for Hx+Ex profiles, there was an overall improvement in diagnosis, particularly for glaucoma. The top 3 subspecialties in overall accuracy were glaucoma; lens diseases; and eyelid, lacrimal, and orbital diseases. In the subspecialties of eye tumors and scleral and uveal diseases, significant differences were observed in the top-ranked diagnosis accuracy for Hx+Ex profiles among GPT-3.5 , GPT-4.0, and residents (4/11, 36% vs 5/11, 46% vs 10/11, 91%; P=.03; and 1/6, 17% vs 5/6, 83% vs 5/6, 83%; P=.02, respectively). Notably, for scleral and uveal diseases, GPT-3.5 demonstrated a lower accuracy of 17% (1/6) both in the top-ranked diagnosis (P=.02) and within the top 3 diagnoses (P=.02) compared to GPT-4.0 (5/6, 83%) and the residents (5/6, 83%; Figure 3).

For Hx+Ex profiles where the medical history provided past diagnoses related to the final diagnosis, a higher top-ranked diagnosis accuracy was observed in the GPT-4.0 model (7/7, 100% vs 55/97, 57%; P=.04). However, case descriptions including past diagnoses unrelated to the final diagnosis and cases requiring ophthalmic examination for a definitive diagnosis did not significantly affect the top-ranked diagnosis accuracy for GPT-3.5, GPT-4.0, and the residents (all P>.21; Table 4).

In the analysis of incorrect top-ranked diagnoses from Hx+Ex profiles, GPT-4.0 exhibited fewer incorrect top-3 diagnoses than GPT-3.5 (25/42, 60% vs 53/63, 84%; P=.005), making partially correct diagnoses with incorrect lesion nature (21/42, 50% vs 7/63, 11%; P<.001). In contrast, GPT-3.5 often made completely incorrect diagnoses about lesion nature (27/63, 43% vs 7/42, 17%; P=.005) and exhibited less precision with no further diagnosis more frequently than GPT-4.0 (22/63, 35% vs 5/42, 12%; P=.009; Table 5).

AI has demonstrated its potential in facilitating accurate patient registration and health care services [13,14]. With iterations, ChatGPT—a chatbot powered by AI—has shown promise for triage in ophthalmic emergencies and in achieving diagnostic accuracy in simulated vignettes [6,15]. Our study focused on investigating the triage and diagnostic value of ChatGPT. To our knowledge, this was the first study to explore the role of ChatGPT in ophthalmic registration. Additionally, we designed 2 types of information inputs, Hx and Hx+Ex profiles, and simultaneously tested the accuracy of leading diagnoses and do-not-miss diagnoses. This approach helped provide a comprehensive understanding of ChatGPT’s performance. While existing studies mainly focus on the application within a single ophthalmic subspecialty [10,11,16,17], another strength of our study was the use of a diverse set of diagnostic cases, covering 12 ophthalmic subspecialties and 104 distinct cases. This breadth enhances the AI system’s evaluation across varied clinical scenarios.

From the patients’ perspective, AI chatbots facilitate ophthalmic triage and appointment [18]. Patients know their own complaints and medical histories well. This knowledge enables them to directly interact with AI chatbots, seeking advice on appropriate ophthalmic subspecialties for registration [4]. Our study used Hx profiles to simulate this self-service interaction, highlighting the practical utility of chatbots in patient-initiated health care navigation. The findings demonstrated that GPT-4.0 directed patients to the correct registration with 78% accuracy, which numerically surpassed the 69% accuracy achieved by medically trained residents. In China, where major tertiary hospitals have implemented web-based registration systems, such as through a WeChat-based medical platform [19], the integration of a user-friendly and accessible AI chatbot significantly streamlines the consultation process for patients, particularly those unsure of which ophthalmic subspecialty to choose. This study provided the first empirical evidence of how AI chatbots can facilitate more accurate and efficient patient registration in clinical settings.

For ophthalmic diagnoses, studies had shown that GPT-4.0 exhibited lower than 50% accuracy in deriving leading diagnoses from complaint records alone [6], while the free version of GPT-3.5 underperforms compared to medical residents [12]. These findings were consistent with our study. In a pilot study by Hu et al [20], GPT-4.0 was tested on its capability to diagnose rare eye diseases, revealing that more comprehensive information provided to the model resulted in considerably more “right” diagnoses. The reduced accuracy when used with limited patient information (such as Hx only) indicated that GPT-4.0 was not yet suitable as a stand-alone diagnostic tool. However, adding details from patient examinations significantly enhanced its performance, making it comparable to that of residents. In real-world clinical settings, where physicians have access to comprehensive case information, GPT-4.0 achieved diagnostic accuracies between 60% to 76%, demonstrating its potential to support clinical decision-making. Similar performance had been observed in uveitis studies, with accuracy rates ranging from 60% to 66% [21,22]. Unlike GPT-3.5, which provided imprecise diagnoses, GPT-4.0, even when erring in its initial diagnosis, tended to include the correct diagnosis within the top 3 suggestions. This illustrated GPT-4.0’s superior information retrieval capabilities and its role as a diagnostic aid [23]. It is expected that with continuous improvements, AI chatbots will play an increasingly vital role in enhancing health care efficiency [24,25].

Our findings revealed that the accuracy of recommendations for ophthalmic subspecialty registration did not consistently correlate with diagnostic accuracy. For example, while the diagnostic accuracy for glaucoma reached 100%, the accuracy of leading patients to register in the glaucoma department was notably low. This discrepancy was likely attributed to the frequently nonspecific initial complaints associated with glaucoma, which generally require additional clinical examinations to establish a definitive diagnosis, such as intraocular pressure, OCT, and visual field tests. Therefore, relying solely on patient complaints and medical histories proved insufficient for accurately guiding patients to the appropriate glaucoma specialty for initial registration. Conversely, when patient histories were supplemented with specific ophthalmic examinations, the accuracy of differential diagnosis improved, thereby enhancing overall diagnostic performance.

Compared to the accuracy of GPT-4.0 in answering ophthalmic questions [26-28] and suggesting surgical plan [29], which can exceed 80%, its performance in registration recommendations and clinical diagnosis was intermediate or inferior. This discrepancy may be attributed to the conservative diagnostic judgment standards and the inclusion of less common diseases. Although the evaluated profiles were derived from real-world clinical reports in the published Chinese SRT database, the inclusion of uncommon and atypical cases was inevitable. Such heightened complexity poses significant challenges not only for AI chatbots but also for residents in the early stages of their medical careers [30]. In real-world clinical settings, the uneven distribution of diseases across various ophthalmic subspecialties typically leads to variability in diagnostic outcomes.

Given that the performance of AI chatbots is contingent upon the volume of information provided, future AI chatbots may still require upgrades and iterations to mitigate information asymmetry between patients and health care professionals, thereby enhancing the delivery of more effective and professional ophthalmic care. For example, AI chatbots specifically designed and trained for ophthalmic care [31,32]; chatbots that can proactively solicit information not provided by end users, similar to the process used by ophthalmologists; and those capable of directly accessing and interpreting imaging data like a multimodal AI chatbot, are essential. Unlike general-purpose AI chatbots such as ChatGPT, Zheng et al [33] have developed a Chinese large language model for ophthalmology using a corpus with extensive clinical vignettes (Hx+Ex). This model demonstrated a diagnostic accuracy of 81.1% across 6 common ophthalmic subspecialties, surpassing the performance of GPT-4.0 (59.6%) in our study. However, the current capability of multimodal GPT-4.0 to diagnose vitreoretinal diseases through the analysis of retinal images remains less than optimal [34].

ChatGPT allows users to customize responses based on personalized prompts, as illustrated in this study by providing 3 differential diagnostic recommendations and the corresponding rationale [35,36]. Research showed that ChatGPT consistently offers a broader range of differential diagnoses than ophthalmology residents [30]; this tendency was also observed in our data collection. The ability to organize key case information through the explanation of diagnostic reasoning not only enhances the knowledge structure of physicians but also underscores the significant educational value of AI chatbots in medical training [37-39]. The proficiency of AI chatbots in responding to ophthalmic examination questions and addressing eye disease queries have been confirmed by recent studies [40-42]. However, it is crucial to note that current AI chatbots do not necessarily replace the clinical judgment of professional ophthalmologists. Their application is still subject to ethical considerations [43,44] and concerns about hallucinations [45]. Physicians responsible for diagnosis need to remain cautious when considering information provided by ChatGPT. The extent to which AI can serve as an adjunct tool in health care still requires further real-world testing.

In clinical settings, ophthalmologists typically rely on direct observation of patients’ symptoms and examinations for intuitive face-to-face diagnoses. However, the text-based format of medical records may not fully reflect ophthalmic residents’ capabilities and might even underestimate them. Additionally, while images are crucial for ophthalmic diagnosis, GPT-4.0’s support for multimodal data still shows suboptimal performance in image-based cases [46]. This explains our decision to exclude images from our evaluation of ChatGPT. However, textual descriptions of these images could have impacted the outcomes. Moreover, although ChatGPT supports multiple languages, differences in language use in diagnosing eye diseases have been observed [11]. Our study, conducted in Mandarin Chinese, may affect the generalization of the results. Despite these limitations, our study contributes to understanding AI’s role as a tool in ophthalmic health care.

Our study showed that GPT-3.5 and GPT-4.0 demonstrated moderate performance in directing patients to appropriate ophthalmic subspecialties for registration. While GPT-3.5 was less effective, GPT-4.0 approached and even numerically surpassed residents in differential diagnosis when presented with patient histories and examination results. AI chatbots merit emphasis for their potential to facilitate patient registration and optimize consultations in ophthalmology. While their diagnostic capabilities could benefit ophthalmologists, integrating them into diagnostic decision-making still requires further validation.