This study was approved by the institutional review board of Hallym University Chuncheon Sacred Heart hospital (approval number: 2022-03-002). This study adhered to the guidelines for developing and reporting machine learning predictive models in biomedical research [17].

This study extends the previous research [8,9] on this topic by constructing a 6-class lesion classification model (Figure 1) and a preneoplastic lesion segmentation model (Figure 2). A CDSS was constructed by combining the established 6-class lesion classification model and the preneoplastic lesion segmentation model with the previously established lesion detection model. All images in the training, internal test, and external test data sets were mutually exclusive.

We extended the data collection process in order to develop the automated lesion classification and lesion segmentation models. The CDSS was constructed using input images that were collected retrospectively. The detailed data collection procedure was described previously [8,9]. In brief, between 2010 and 2017, we enrolled consecutive patients with any type of gastric neoplasm discovered during upper gastrointestinal endoscopy and pathologically confirmed at the Chuncheon Sacred Heart hospital.

We enrolled all patients diagnosed with atrophy or IM during upper gastrointestinal endoscopy at Hallym University Chuncheon Sacred Heart hospital between 2019 and 2021 to add preneoplastic conditions to the baseline input data. These images were classified as either atrophy or IM. To reduce interobserver variability and ensure accurate categorization, all enrolled images were cross-checked by 2 expert endoscopists (CSB and EJG). Discordantly categorized images were resolved through discussion. The “normal” category was created using the same procedure as described above but without atrophy or IM. Endoscopic images of patients found to be free of gastric neoplasm, atrophy, or IM during upper gastrointestinal endoscopy at Hallym University Chuncheon Sacred Heart hospital between 2019 and 2021 were collected in JPEG format, with a minimum resolution of 640 x 480 pixels, from the in-hospital database [8,9].

Finally, 11,868 white-light images were enrolled and randomly divided into training (n=9999) and internal test (n=1869) data sets. Table 1 describes the detailed distribution of the input images.

Endoscopic still-cut images contain “noise” information, such as the date of the examination and the patient’s name, age, gender, or identification number. The semantic segmentation of the input images can remove this noise information from the images and improve the training efficacy in the establishment of deep-learning models. For the preprocessing of the included images, we used a modified U-Net++ [18] convolutional neural network model (an edge-smoothing algorithm was added in the U-Net++, and the backbone convolutional neural network was Densenet121), as previously described [11]. Data augmentation methods were used, including rotation (–10° to +10°), horizontal or vertical flipping of included images, and image normalization with linear transformation in terms of the red, green, and blue channels [11].

All deep-learning models were built with Python (version 3.10; Python Software Foundation) and the SQLite 3.41.2 database management system (SQLite). The convolutional neural network models were implemented using the TensorFlow framework [19]. EfficientNet-B3 [20] was used to create the 6-class lesion classification model, while UNet++ with EfficientNet-B3 was used to create the lesion segmentation model. Additionally, 4 RTX 3090ti graphics processing units (MSI), AMD Ryzen Threadripper PRO 5975WX 32-Core central processing units, and 512 GB RAM (Samsung) were included in the training system.

In order to maintain a seamless evaluation of the gastric mucosa without interfering with the endoscopic inspection, as previously explained, the CDSS was designed to give endoscopists additional information [11]. The primary deep-learning server and local computer system were first built independently. We linked the primary endoscopic system to a local computer with an additional display. The endoscope monitor image was entered into this local computer system, which then executed real-time detection tasks, asked the deep-learning server to classify the freeze images (called the motion freeze function), and displayed the results. The segmentation results were shown on the monitor after the atrophy or IM was identified and classified. The user interface for the established CDSS is described in Figure 3.

This study used a previously established lesion detection model [11]. Briefly, YOLOv3 [21], which was modified with EfficientNet-B0 structure, was implemented for the establishment of the model, and 2653 white-light images were enrolled for the training data set [11]. The detection model immediately transfers the detected gastric lesions to the classification model, where they are automatically classified into 6 classes (histologic diagnosis). When the CDSS detects a lesion, the frame containing the lesion is automatically examined by the classification model, yielding findings for 6 classes. The segmentation results are shown on the monitor after the atrophy or IM has been identified and classified (the detected and classified atrophy and IM are the input data of the segmentation model; Figure 4) The CDSS analyzes the dependency of successive frames (calculating the difference of brightness and contrast in successive frames) in order to detect the doctor’s intention (motion freeze function). The mean response time was 0.3 seconds between the classification request and the display of the findings on the monitor.

The video frame is encrypted and transferred with Academy Research Institute Agency encryption, which is a common cryptographic method by the Korean Agency for Technology and Standards used when a local computer requests a classification task from the main server to ensure anonymization. In order to prevent leakage and forgery or modification, medical information is also communicated between the deep-learning server and the local computer using the HTTP secure communication protocol and the secure sockets layer security protocol.

The primary outcomes included the classification model’s lesion classification accuracy (6 classes: AGC, EGC, dysplasia, atrophy, IM, and normal) and atrophy and IM lesion segmentation rates for the segmentation model. Diagnostic accuracy was defined as true positive + true negative / (true positive + false positive + true negative + false negative). Mean intersection over union (IoU) was defined as (A∩B) / (A∪B), and mean dice coefficient (F1 score) was defined as (2 × |A∩B|) / (|A|+|B|), where A signifies the predicted set of pixels (predicted segmentation map) and B is the ground truth of the object to be found in the image. Segmentation rate was defined as true positive / (true positive + false negative). For a positive image, true positive was defined as an IoU of the artificial intelligence (AI) prediction result mask greater than the threshold (0.5) and false negative was defined as an IoU of the AI prediction result mask less than the threshold (0.5).

To validate the performance of lesion classification accuracy, the following tests were performed (Table 1):

To estimate the clinical utility of the CDSS, 2 additional experienced endoscopists were invited to perform a blind test with the same data set. To compare the diagnostic performances of the CDSS and endoscopists, we performed Fisher exact tests. In this study, a P value <.05 (2-tailed) was adopted as the threshold of statistical significance. The analyses were performed using SPSS version 24.0 (SPSS).

A total of 13,150 endoscopic still-cut images were included in the study, comprising the training data set (n=9999), the internal test data set (n=1869), and the external test data set (n=1282). Of the images included in the training and internal test data sets (n=11,868), those with IM made up the largest proportion (n=3622; 30.5%), followed by normal (n=2990; 25.2%), atrophy (n=2603; 21.9%), EGC (n=1067; 9%), dysplasia (n=932; 7.9%), and AGC (n=654; 5.5%) images.

For the external data set, which comprised 1282 images collected from another institution, the proportion of normal images was the largest at 19.1% (n=245), followed by atrophy images (n=243; 19%), IM images (n=235; 18.3%), EGC images (n=190; 14.8%), AGC images (n=186; 14.5%), and dysplasia images (n=183; 14.3%). These proportions reflect the distinctive characteristics of the institution from which the images were collected.

A total of 522 consecutive screening endoscopy images were enrolled for prospective real-clinic evaluation of the established CDSS. The proportion of images with atrophy was the largest (n=252; 48.3%), followed by those with IM (n=134; 25.7%), dysplasia (n=8; 1.5%), EGC (n=4; 0.8%), and AGC (n=3; 0.6%). Details of the quantity of images and the distribution of categories are presented in Table 1.

Multimedia Appendices 1 and 2 describe the schematic framework of the established lesion classification and lesion segmentation models. The overall 6-class lesion classification accuracy (95% CI) in the internal test data set of the classification models was 90.3% (89%-91.6%). The accuracy for each category was as follows: atrophy was 96.5% (94.9%-98.1%), IM was 90.4% (87.4%-93.4%), EGC was 78.2% (72.8%-83.6%), AGC was 86.2% (80.3%-92.1%), dysplasia was 75.6% (69.6%-81.6%), and normal was 97% (95.4%-98.6%). The detailed classification performance of the established CDSS is described in Tables 2 and 3, and a confusion matrix for the internal test classification performance is presented in Multimedia Appendix 3. The detailed hyperparameter information for the established lesion classification model was as follows: weight initialization with ImageNet; augmentation, horizontal/vertical flip, rotate (−10° to +10°); input size, 300 x 300; batch size, 32; learning rate, 1 × 10^-04; epoch, 100; dropout, 0.2; optimizer, SGD; loss function, cross-entropy loss; and learning rate scheduler, cosine decay.

In terms of the lesion segmentation rate, the CDSS demonstrated a segmentation rate of 93.4% (95% CI 92.4%-94.4%) for atrophy or IM lesion segmentation in the internal test data set (Table 4). Multimedia Appendix 4 shows the representative segmentation results of atrophy and IM locations that correspond to the ground truth area. The detailed hyperparameter information for the established lesion segmentation model was as follows: weight initialization with ImageNet; augmentation, horizontal/vertical flip, rotate (−10° to +10°); input size, 352 x 352; batch size, 16; learning rate, 1 × 10^-04; epoch, 100; dropout, 0.2; optimizer, Adam; loss function, binary cross-entropy + IoU; and weight decay, 1 × 10^-04.

For performance validation with the external test data set, the CDSS achieved an overall accuracy (95% CI) of 85.3% (83.4%-97.2%). The accuracy for each category was as follows: atrophy was 95.3% (92.6%-98%), IM was 89.3% (85.4%-93.2%), EGC was 76.3% (70.3%-82.3%), AGC was 82.8% (77.4%-88.2%), dysplasia was 74.9% (68.6%-81.2%), and normal was 88.2% (84.2%-92.2%). The detailed performance of the established CDSS is described in Table 2, and a confusion matrix for the external test performance is presented in Multimedia Appendix 5.

For performance validation with the prospective real-clinic evaluation data set, the CDSS achieved an overall accuracy of 89.3% (86.6%-92%). The accuracy for each category was as follows: atrophy was 92.1% (88.8%-95.4%), IM was 95.5% (92%-99%), EGC was 50% (1%-99%), AGC was 100% (100%-100%), dysplasia was 75% (45%-100%), and normal was 78.5% (71.2%-85.8%). The detailed performance of the established CDSS is described in Table 2, and a confusion matrix for the prospective real-clinic evaluation performance is presented in Multimedia Appendix 6.

To estimate the clinical utility of the CDSS, 2 experienced endoscopists (HMJ and GHB) were invited to perform a blind test with the same data set. For the first endoscopist, the overall accuracy was 87.5% (84.7%-90.3%), and the accuracy for each category was as follows: atrophy was 86.1% (81.8%-90.4%), IM was 80.6% (73.9%-87.3%), EGC was 75% (32.6%-99%), AGC was 100% (100%-100%), dysplasia was 62.5% (29%-96%), and normal was 100% (100%-100%). For the second endoscopist, the overall accuracy was 90.8% (88.3%-93.3%), and the accuracy for each category was as follows: atrophy was 91.3% (87.8%-94.9%), IM was 84.4% (73.9%-94.8%), EGC was 100% (100%-100%), AGC was 100% (100%-100%), dysplasia was 87.5% (64.6%-100%), and normal was 90.9% (85.8%-96%). The Cohen κ for the interrater reliability between the 2 endoscopists was 0.83. There was no significant difference in the overall accuracy between the invited endoscopists and the established CDSS in the prospective real-clinic evaluation data set (P=.23).

Multimedia Appendix 7 shows representative examples of the CDSS for the classification and segmentation of gastric atrophy and IM. The established CDSS detected, classified, and segmented gastric atrophy and IM on the angle and lesser curvature sides of the body. A small subepithelial tumor was detected and correctly classified as normal.

This study established a CDSS that accurately classifies all stages of gastric carcinogenesis and confirmed that the diagnostic accuracy of 6 classes is potentially accurate for real-world clinical applications. Our research group previously established a classification model with 4 or 5 classes for the diagnosis of detected lesions during upper gastrointestinal endoscopy [8,11]. Normal, dysplasia (low-grade or high-grade), EGC, and AGC were the diagnostic categories. However, preneoplastic conditions, such as gastric atrophy and IM, are important categories in which hidden neoplastic lesions or potential lesions for the development of neoplastic lesions are implicated. Our previously established CDSS did not include these categories and considered them to be normal [8,11]. In general, as the number of target classification categories increases, classification accuracy decreases. However, when compared to our other models, our established CDSS performed similarly or even better in terms of diagnostic performance. This is most likely due to an increase in the number of training data images.

The presence of external validation is one of the strengths of this study. Even the majority of US Food and Drug Administration–approved medical AI models lack external validation, which ensures the established model’s generalizability [22]. The majority of currently established models are trained on data from their own institution. As a result, internal test performance is good; however, external test performance suffers due to the “data set shift” phenomenon (differences in data settings, patient characteristics, and so on) [14-17,22]. Our established model also demonstrated lower external test performance when compared to internal test performance; however, the magnitude of the performance decrease was minor, and external test accuracy was still high enough to potentially be used in real-clinic applications.

We also performed prospective real-clinic application performance validation to decrease the gap between theory and practice. When compared to the training data set, real-world clinical practice has distinct characteristics. For the training data set, we included an even number of images in each category; this was because class distribution influences the overall performance of the established model [16]. The inclusion of a class with an insufficient number of images negatively impacts the performance for that class and the established model as a whole; this is known as spectrum bias [14-16]. However, clinical practice differs. The majority of screening endoscopies are classified as “normal,” with only a small number of patients being diagnosed with neoplastic conditions. As a result, neoplastic lesion–focused AI diagnosis models may be clinically ineffective. The current CDSS was developed using the previously established 4-class diagnosis model of gastric neoplasms [11]. We added the ability to make more precise diagnoses of preneoplastic or nonneoplastic conditions. As a result, the current CDSS may be useful in real-world applications. Prospective real-clinic application performance revealed potentially high accuracy not only for gastric atrophy and IM lesions, but also overall accuracy. There was also no significant difference in the overall accuracy between the experienced endoscopists and the established CDSS.

Target conditions and training data sets distinguish our established CDSS from previously reported models. Guimarães et al [23] created an atrophy detection model that outperformed expert endoscopists with 93% accuracy in an independent data set. Zhang et al [24] also developed a lesion detection model for gastric atrophy that was 94% accurate. Zhao and Chi [25] also developed a gastric atrophy lesion detection model and demonstrated that using this model improved gastric atrophy diagnosis rates when compared to not using the model. Luo et al [26] created a lesion detection model for gastric atrophy that outperformed endoscopists in detection performance. However, the utility of this type of data set in real-world clinical practice is limited. These studies generated a training data set that included only atrophy and no atrophy categories. The categories of gastric cancer, normal, and IM were not considered in these studies. Because they only allow for discrimination between atrophy versus no atrophy, the clinical utility of these models is limited. Siripoppohn et al [27] developed a real-time semantic segmentation model for IM; despite its excellent diagnostic performance, only IM could be segmented in real time.

Various conditions are included in the “normal” category of upper gastrointestinal endoscopy, which excludes neoplastic lesions. It is common to have gastritis without atrophy or IM. These include mucosal edema or exudates, hemorrhagic spots, erosions, hyperemic mucosa, and other conditions. On the other hand, there are also more complicated benign conditions, such as benign ulcers or ulcer scars, subepithelial tumors, xanthomas, nonneoplastic polyps (eg, fundic gland or hyperplastic polyps), or angiodysplasias. To complement the endoscopic inspection without interfering with the operator’s examination, our CDSS was established. This CDSS detects stomach lesions automatically and provides additional information (histologic prediction) via an alarm sound and a lesion detection box. The effect of this system varies depending on the level of expertise of the endoscopist, and previous research has shown that endoscopists with a certain level of expertise benefit the most from this CDSS [10].

Multimodal AI foundation models, including large language models, are becoming more popular. This type of model may perform a more sophisticated practice involving decision making. Using these models, it would be possible to create a multimodal CDSS capable of not only supplementing endoscopic examinations but also writing endoscopic reports and providing personalized explanations or recommendations [22]. CDSS detection and classification of all gastric lesions, including benign conditions, may not be required for expert endoscopists; however, trainee or novice endoscopists may require the assistance of this CDSS to ensure a consistent level of performance for the detection or classification of lesions during endoscopic examination. Creating such complex models would be possible with the help of multimodal foundation AI models.

Despite the benefits of our established CDSS, an inherent flaw arising from the study design and retrospectively collected data was discovered in this study. The training data set was obtained from a single institution, which may indicate a bias in selection or spectrum. Because of the unique characteristics of each institution’s patients, medical AI models developed from a single institution typically have limitations for widespread implementation, highlighting the importance of external testing. To compensate for this flaw, we used rigorous prospective validations using images from another institution. In addition, we validated the clinical utility through prospective real-clinic application validation.

In conclusion, the CDSS demonstrated high performance in terms of computer-aided diagnosis of all stages of gastric carcinogenesis and demonstrated real-world application potential.