This study was conducted in accordance with the 1964 Declaration of Helsinki guidelines. The Ethics Committee of Hangil Eye Hospital approved the research protocol (IRB 21018) and its implementation and waived the requirement for informed consent as this study was retrospective and observational in nature and used medical records to extract the required data.

To investigate whether each component of the proposed system can assist in improving the diagnostic performance of ophthalmologists, a web-based experiment was conducted in which each participant was instructed to classify CSC subtypes from a given SD-OCT image. The experimental procedure comprised 3 steps (Figure 2). In the first step, observers had to identify the possible CSC subtype based on the SD-OCT image. The observers diagnosed retinal disease without artificial intelligence assistance (ie, No AI). In the subsequent step, an AI probability panel was provided to the observers (ie, artificial intelligence assistance with a probability score [AI Prob]). The AI probability panel shows the probability score of each retinal disease (acute or chronic). At the end of the step, both the AI probability panel and AI evidence heatmap panel were added to the system to provide a visual explanation to the observers (ie, artificial intelligence assistance with a probability score and visual evidence heatmap [AI Prob+Evid]).

In each step, all observers had to determine whether the given OCT image reflected acute or chronic CSC by selecting a button on the web system. The same OCT image was used in the 3 steps. As 100 SD-OCT images were used in our experiment, each participant assessed 300 cases (ie, 3 steps × 100 images) in total. The 100 images were randomly extracted from the test set that was not used to train our model. The step-by-step user interface for the observer performance test is illustrated in Multimedia Appendix 4.

The study recruited 66 participants, including 36 retina and 30 nonretina specialists. The retina specialists were medical doctors who had completed 1-2 years of the retina fellowship training program. In contrast, nonretina specialists were board-certified ophthalmologists who were not specialized in the retina. The detailed information of the 66 participants is summarized Multimedia Appendix 5.

Receiver operating characteristic analysis was conducted to evaluate the performance of the proposed model and ophthalmologists in classifying the CSC subtypes. Thereafter, the receiver operating characteristic curve with the true-positive and false-positive rates was plotted to measure the area under the receiver operating characteristic curve (AUROC) score. Friedman [28] test, followed by the Wilcoxon signed-rank test, was used to quantify the differences among the 3 different conditions (ie, No AI, AI Prob, and AI Prob+Evid) [29]. A 2-tailed t test was used to compare diagnostic performance between the nonretina and retina specialists.

The proposed model exhibited high accuracy, sensitivity, and specificity values of 96.3%, 97.1%, and 95.7%, respectively (n=163). The model achieved 98.4% (n=163) of the AUROC which outperforms Resnet-50 (n=163; AUROC 87.9%) and VGG-19 (n=163; AUROC 96.1%). The model failed to accurately predict 6 cases only in our test set. Importantly, the training AUROC of 95.6% (n=163) indicates that our model strikes a balance between avoiding overfitting and underfitting, further affirming its reliability.

The diagnostic performance of the AI-CAD system was compared with that of ophthalmologists. The AUROCs were calculated to evaluate the AI-CAD and human predictive abilities for 100 images randomly extracted from the test set. The AI-CAD system (AUROC 99.5%; n=100) outperformed both retina (AUROC 92.1%; n-36) and nonretina (AUROC 87.8%; n=30) specialists.

The retinal-disease detection performance of the 36 retina and 30 nonretina specialists were evaluated under 3 different conditions (ie, No AI, AI Prob, and AI Prob+Evid; Table 1). The results of the Friedman test followed by the Wilcoxon signed-rank test revealed significant differences in diagnostic performance among the 3 different conditions for retina (statistic=59.5, df=2; P<.001) and nonretina (statistic=44.4, df=2; P<.001) specialists. In particular, the retina specialists who were provided with the AI probability panel and AI evidence heatmap panel (ie, AI Prob+Evid) achieved the highest mean diagnostic performance (AUROC 95.8%, 95% CI 0.948-0.969; n=36) compared with those subjected to other conditions (No AI: 0.921, 95% CI 0.907-0.935; P<.001; and AI Prob: 0.956, 95% CI 0.946-0.967; P<.05). The nonretina specialists also displayed their best performance (0.929, 95% CI 0.913-0.946) with numerical and visual information compared with when they were subjected to other conditions (No AI: 0.878, 95% CI 0.860-0.895; P<.001; and AI Prob: 0.922, 95% CI 0.905-0.940; P<.001).

This study investigated how the AI-CAD system can help nonretina specialists detect retinal diseases. The t test results revealed no significant differences in AUROC between the nonretina specialists supported by the AI-CAD system (AI Prob: 0.922, 95% CI 0.905-0.940; P=.88; and AI Prob+Evid: 0.929, 95% CI 0.913-0.946; P=.42) and retina specialists not supported by the AI-CAD system (No AI: 0.921, 95% CI 0.907-0.935). This finding demonstrates that nonretina specialists can achieve expert-level diagnostic performance with the support of the proposed AI CAD system (Figure 3).

To evaluate the proposed system’s positive effect in assisting ophthalmologists, the number of positive (ie, false negative to true positive and false positive to true negative) and negative (ie, true positive to false negative and true negative to false positive) changes that could be observed between the No AI and AI Prob+Evid conditions were recorded. Chronic CSC was set as the positive class. Overall, 42.8% (307/718) of the misclassified cases under the No AI condition were accurately classified in the AI Prob+Evid condition (Table 2). In particular, 106 false-negative cases turned into true positives after using the proposed AI-CAD system, implying that the proposed system is useful for ophthalmologists in distinguishing between acute and chronic CSC.

This study proposed the development of an AI-CAD system to assist ophthalmologists in distinguishing chronic from acute CSC. In particular, the proposed system provides (1) the probability of retinal disease and (2) visual evidence to effectively assist ophthalmologists in their clinical decisions. To evaluate the effectiveness of the proposed AI-CAD system in enhancing ophthalmologists’ clinical decision-making, a within-subject user study involving 66 ophthalmologists was conducted. The extensive experiments demonstrated that the proposed AI-CAD system effectively assists ophthalmologists in improving their diagnostic performance for retinal disease.

The proposed deep neural network in the AI-CAD system achieved a high retinal-disease detection performance of 99.5% (n=100) of the AUROC, outperforming all 66 ophthalmologists who participated in the experiment. The high performance of the proposed AI-CAD model implies that it can lessen the heavy workloads and reduce potential errors by clinicians [8,9]. The quality of clinical services can be improved by using a deep learning model that guarantees consistent and high-level detection performance for retinal disease.

The observer performance test revealed that the proposed AI-CAD system can effectively help ophthalmologists diagnose retinal disease. The experimental results revealed that the diagnostic performance of the retina specialists and nonretina specialists increased by 3.5% (n=36) and 4.4% (n=30), respectively (Table 1). This signifies that the proposed system successfully improved the ability of ophthalmologists to detect retinal disease. In particular, the performance difference between cases with and without the AI-CAD system was higher in the nonretina specialist group (n=30; 4.4%) than that in the retina specialist group (n=36; 3.5%), implying that the nonretina specialist group tended to receive support more than the retina specialist group. This finding is consistent with that of a prior study, which revealed that relatively less-skilled physicians exhibited greater improvement in the detection of pulmonary disease compared with that of skilled physicians [6,7]. Moreover, ophthalmologists achieved an even higher AUROC (retina specialists: 0.958; nonretina specialists: 0.929) if AI diagnosis information was availed with its visual explanation (gradient-weighted class activation mapping). This indicates that providing probability scores with a visual explanation is more useful for ophthalmologists than simply displaying the probability scores alone, thereby exhibiting consistency with prior work that revealed the usefulness of visual modality in detecting diseases [7].

Deploying a clinical decision support system (CDSS) in real-world health care settings presents a set of practical challenges. Chief among these concerns is the system’s susceptibility to errors, which can undermine trust in AI-driven solutions. To address this, a dynamic training model becomes indispensable. In the realm of academia, researchers have embraced the “Human-In-the-Loop” paradigm to tackle this issue [30-32]. This approach involves the seamless integration of human oversight and intervention into the CDSS’s decision-making processes. By empowering human experts to review and amend the system's outputs, we expedite the identification and rectification of errors. Consequently, this iterative feedback mechanism bolsters the CDSS's trustworthiness and reliability in real-world applications, bringing it into closer alignment with user expectations and requirements.

Another pivotal concern pertains to the system's security. Safeguarding patient data and upholding the integrity of the CDSS is paramount within the health care domain. This demands the implementation of robust encryption, stringent access controls, and regular security audits to protect sensitive information and prevent unauthorized access or data breaches. Furthermore, continuous monitoring and timely updates to the CDSS are essential for addressing emerging security threats and vulnerabilities, ensuring a high level of security in real-world applications.

By concurrently addressing error mitigation and security, CDSS developers and health care professionals can collaborate in creating a more dependable and trustworthy system that serves the best interests of both patients and medical practitioners.

This study has some limitations. First, all images were acquired from a single OCT device located at a single academic center. Although the data set was sufficient to train and validate the proposed model for distinguishing between CSC subtypes, external validation with a different center is needed. Second, experiments were conducted using the web-based AI-CAD system developed in this study. Thus, the environment was relatively different from that of actual clinical practice. However, we attempted to design and develop a user-friendly AI-CAD system under the supervision of retina specialists. Third, considering that the model's training data comprises images taken exclusively by the Heidelberg Spectralis device (Heidelberg Engineering Inc), its performance might be insufficient when dealing with images from different devices. Future studies should prioritize (1) extending the proposed AI-CAD system to other retinal diseases, such as AMD and diabetic retinopathy; (2) developing strategies to improve the reliability of doctors when using the AI-CAD system; and (3) exploring the application of transfer learning techniques to address the challenges arising from variations in devices.

This study has several implications. First, to the best of our knowledge, this study is the first attempt to develop and evaluate an AI-CAD system for the detection of retinal disease using OCT. Prior studies have developed AI-CAD systems for the detection of pulmonary disease and evaluated their effectiveness [6,33,34]. However, minimal attention has been focused on the application of an AI-CAD system for the diagnosis of retinal diseases, such as CSC and AMD. In this study, an AI-CAD system that can assist ophthalmologists in identifying retinal diseases was developed and its usefulness in detecting retinal disease was evaluated.

Second, the proposed AI-CAD system is potentially useful for small or local medical care centers where retina specialists are unavailable. Unlike in large-scale medical care centers or hospitals, retina specialists are rarely found in small or local centers. Diagnosing subtypes of retinal diseases (ie, acute vs chronic CSC [8] and polypoidal choroidal vasculopathy vs retinal angiomatous proliferation [10]) requires more elaborate expertise than simply screening abnormal cases (ie, normal vs CSC [8]), and the proposed AI-CAD system exhibits higher performance than that of retina specialists with over 10 years of experience, implying that the proposed system potentially plays an important role in such cases. In CSC, assessing the chronicity of the disease at the time of diagnosis is crucial for selecting an appropriate course of treatment or forecasting its prognosis [8,15,22]. Acute CSC typically follows a self-limiting natural course, whereas chronic CSC with or without sustained SRF may be associated with irreversible vision loss or may require active intervention, such as intravitreal antivascular endothelial growth factor injections or photodynamic therapy, all of which are intended to prevent long-term visual loss that can lower the patient’s quality of life. Further, on comparing diagnostic performance between nonretina and retina specialists, the experimental results of this study demonstrate that nonretina specialists can achieve retina specialist-level performance with the support of the proposed AI-CAD system (Figure 3). This implies that the proposed system can alleviate the heavy workload of ophthalmologists who have expert-level diagnostic performance and facilitate the decision-making process of less-skilled ophthalmologists (nonretina specialists) by improving their diagnostic performance.

To the best of our knowledge, this study is the first attempt to design, develop, and evaluate an AI-CAD system for the detection of retinal disease using OCT. First, an AI-CAD system was developed with a high-performance deep learning model. Thereafter, an observer performance test was conducted with the proposed system to determine the ability of the system to assist ophthalmologists in diagnosing retinal diseases. The results indicated that the proposed AI-CAD system can provide retinal expert-level diagnostic performance and help ophthalmologists improve their diagnostic performance in detecting CSC subtypes. Thus, the proposed AI-CAD system can alleviate the heavy workload of ophthalmologists and help in the decision-making process involved in detecting CSC subtypes. As a base study, this study demonstrates the usefulness and effectiveness of using an AI-CAD system in detecting retinal diseases, particularly CSC subtypes. In the future, the proposed AI-CAD system may be easily extended to the detection of other retinal diseases, such as AMD, diabetic retinopathy, and branch retinal vein occlusion.