This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.
Recently, the analysis of endolymphatic hydropses (EHs) via inner ear magnetic resonance imaging (MRI) for patients with Ménière disease has been attempted in various studies. In addition, artificial intelligence has rapidly been incorporated into the medical field. In our previous studies, an automated algorithm for EH analysis was developed by using a convolutional neural network. However, several limitations existed, and further studies were conducted to compensate for these limitations.
The aim of this study is to develop a fully automated analytic system for measuring EH ratios that enhances EH analysis accuracy and clinical usability when studying Ménière disease via MRI.
We proposed the 3into3Inception and 3intoUNet networks. Their network architectures were based on those of the Inception-v3 and U-Net networks, respectively. The developed networks were trained for inner ear segmentation by using the magnetic resonance images of 124 people and were embedded in a new, automated EH analysis system—inner-ear hydrops estimation via artificial intelligence (INHEARIT)-version 2 (INHEARIT-v2). After fivefold cross-validation, an additional test was performed by using 60 new, unseen magnetic resonance images to evaluate the performance of our system. The INHEARIT-v2 system has a new function that automatically selects representative images from a full MRI stack.
The average segmentation performance of the fivefold cross-validation was measured via the intersection of union method, resulting in performance values of 0.743 (SD 0.030) for the 3into3Inception network and 0.811 (SD 0.032) for the 3intoUNet network. The representative magnetic resonance slices (ie, from a data set of unseen magnetic resonance images) that were automatically selected by the INHEARIT-v2 system only differed from a maximum of 2 expert-selected slices. After comparing the ratios calculated by experienced physicians and those calculated by the INHEARIT-v2 system, we found that the average intraclass correlation coefficient for all cases was 0.941; the average intraclass correlation coefficient of the vestibules was 0.968, and that of the cochleae was 0.914. The time required for the fully automated system to accurately analyze EH ratios based on a patient's MRI stack was approximately 3.5 seconds.
In this study, a fully automated full-stack magnetic resonance analysis system for measuring EH ratios was developed (named INHEARIT-v2), and the results showed that there was a high correlation between the expert-calculated EH ratio values and those calculated by the INHEARIT-v2 system. The system is an upgraded version of the INHEARIT system; it has higher segmentation performance and automatically selects representative images from an MRI stack. The new model can help clinicians by providing objective analysis results and reducing the workload for interpreting magnetic resonance images.
Although many medical fields have been developed over the past few decades, medical imaging techniques, such as computed tomography and magnetic resonance imaging (MRI), have advanced greatly [
Ménière disease is a multifactorial disorder with typical symptoms, such as recurrent vertigo attacks, fluctuating hearing loss, tinnitus, and sensations of ear fullness. The prevalence of Ménière disease varies by region and study, but the estimated prevalence ranges from 30 to 150 patients per 100,000 people [
A protocol for image-based EH analysis was suggested in a previous study [
The need for automated analyses has emerged because conventional quantitative analyses require more time and effort than typical image interpretation processes. The automatic measurement of EH ratios via MRI was proposed based on the deep learning approach in our previous study [
The proposed inner-ear hydrops estimation via artificial intelligence-version 2 (INHEARIT-v2) framework. EH: endolymphatic hydrops; HYDROPS-Mi2: hybrid of the reversed image of the positive endolymph signal and native image of the positive perilymph signal image multiplied by T2-weighted magnetic resonance cisternography; MRC: magnetic resonance cisternography.
Two data sets were used in this study—one for the deep learning–based training models and another for validation. For the training models, the magnetic resonance images of 124 patients (males: n=57; females: n=67; age: mean 49.3 years, SD 16.02 years; range 17-76 years) who participated in our previous INHEARIT study were used [
We scanned all patients’ inner ears with a 3.0-T MRI device (MAGNETOM Skyra; Siemens Medical Solutions). Gadobutrol (gadolinium-DO3A-butriol; Gadovist 1.0), which was used as the contrast agent, was intravenously administered to patients before MRI. Different magnetic resonance scans were made by using heavily T2-weighted magnetic resonance cisternography (MRC) to distinguish the outlines of organs. Positive perilymph images (PPIs) and positive endolymph images (PEIs) were used to evaluate EHs. The HYDROPS images were obtained by subtracting the number of PEIs from the number of PPIs. MRC images, PPIs, and PEIs had identical fields of view, matrix sizes, and slice thicknesses [
One neuroradiologist and one neuro-otologist independently evaluated the MRC images. Representative slices of 4 classes—the left cochlea and vestibule and the right cochlea and vestibule—were selected, and their contours were annotated manually. The main slices of a cochlea or vestibule and their regions of interest (ROIs) were chosen based on the following criteria. For the cochlear ROI, among all MRI slices in which the basal, middle, and apical views of organs were visible, the slice with the largest modiolus height was chosen. For the vestibular ROI, the lowest slice, in which the lateral semicircular canal ring was obvious in images that were rotated by more than 240°, was chosen, and images of the ampulla were excluded [
To segment the ROIs, 2 deep learning models were developed based on the architectures of the Inception-v3 [
The physician-selected main slice image and the previous and next images were loaded simultaneously from full-stack MRC images for model training. The cross-sectional area of target inner ear structures is relatively small in a whole-brain image. Since this can cause a class imbalance problem, image patches of 100×100 pixels were acquired from each left and right reference point. This process is shown with the 3intoUNet network in
The developed models were trained with a selectively annotated data set of 124 subjects. Afterward, the models were fine-tuned with both the selectively annotated and fully annotated data sets, per the optimum curriculum learning strategy for the segmentation of weakly annotated data [
Fivefold cross-validation was conducted, and segmentation performance was evaluated by measuring the intersection over union (IoU) between the ground truth areas (clinician-annotated region) and prediction areas (model-based, automatically determined region). The IoU was calculated as follows:
In equation 1,
Once model training was completed, the model was tested with an additional data set of unseen magnetic resonance images. As shown in
In equation 2,
The EH ratio was estimated by using a HYDROPS-Mi2 image, which is an image that is generated via the pixel-wise multiplication of HYDROPS and MRC image signals [
The above equation (equation 3) can be restated as follows:
In equation 4,
The full-stack magnetic resonance image segmentation analysis process for the test data set. A: A full stack of MRC images and their segmentation results. B: Representative slices and the segmentation results of each class. AI: artificial intelligence; MRC: magnetic resonance cisternography.
The agreement between the values calculated by experts (the neuroradiologist and the neuro-otologist) and the INHEARIT-v2–predicted values was measured by using a single-score intraclass correlation coefficient (ICC), which was based on a two-way model and Pearson correlation coefficient. The analyses were performed by using R software (The R Foundation) [
The segmentation performance of the proposed networks—3into3VGG, 3into3Inception, and 3intoUNet. These networks were based on the Visual Geometry Group-19, Inception-v3, and U-Net networks, respectively. The results are represented as the average intersection over union values from the fivefold cross-validation with SDs.
Networks | IoU,a mean (SD) |
3into3VGG | 0.761 (0.036) |
3into3Inception | 0.784 (0.025) |
3intoUNet | 0.811 (0.032) |
aIoU: intersection over union.
The data set of magnetic resonance images that were collected from 60 new participants (males: n=19; females: n=41; age: mean, 47.1 years, SD 15.27 years; range 21-68 years) for an additional test consisted of 33 subjects with definite Ménière disease (unilateral or bilateral), 17 subjects with sensorineural hearing loss and vertigo, and 10 normal subjects without any symptoms.
System validation was performed on the full-stack images of the subjects. Each image consisted of 104 MRC axial-view image slices, and 3 sequential slices—the main axial-view image slices and the previous and next slices—were fed into the trained model as an input. The model automatically segmented the organs by simultaneously analyzing the three input images and generated a segmentation result as an output. A total of 102 segmented output images were acquired from a subject's stack, and 1 image from each class was selected as a representative result.
With regard to the total 240 image slices of the target organs of 60 subjects, the distance (ie, the number of slice indices) between the manually selected image slices and system-selected image slices was 0 for 105 cases (43.8%), 1 in 233 cases (97.1%), and 2 in 240 cases (100%), as shown by the graph in
Performance of the inner-ear hydrops estimation via artificial intelligence-version 2 (INHEARIT-v2) system in finding the representative slice index in the magnetic resonance image stacks of 240 target organs of 60 subjects.
HYDROPS-Mi2 images were generated at the selected slice index, and EH ratios were calculated for all additional test cases. A case example of segmentation and EH visualization is represented in
Visualization of an endolymphatic hydrops in a left cochlea. A: Magnetic resonance cisternography patch (left ear). B: Left cochlea segmentation result. C: HYDROPS-Mi2 patch (left ear). D: Endolymphatic hydrops. E: Endolymphatic hydrops in the left cochlea. HYDROPS-Mi2: hybrid of the reversed image of the positive endolymph signal and native image of the positive perilymph signal image multiplied by T2-weighted magnetic resonance cisternography.
The agreement between the physician-calculated and INHEARIT-v2–estimated EH ratios was calculated. The average ICC value for all cases was 0.941; the average ICC of the vestibules was 0.968, and that of cochleae was 0.914 (
The correlation between physician-calculated (ground truth) EH ratios and those predicted by the proposed platform (based on the segmentation area). EH: endolymphatic hydrops.
Due to the development of MRI machines, data and image processing technology, big data, and cloud systems, the role of artificial intelligence in the medical field will likely increase with time. Recently, algorithms have begun to help clinicians in real-world clinics and have been used to predict clinical outcomes that are useful for health care systems [
Since the mid-2000s, the analysis of EHs by using 3T MRI has been widely accepted as a useful method for the diagnosis of Ménière disease among various groups [
Recent medical image studies have reported the development of fully automated analytic systems. The implemented tools were designed to provide end-to-end workflows to minimize the amount of human intervention during analysis [
The results of our study substantially complemented the limitations of our previous study [
Another technical improvement over our previous study is that in this study, various deep learning models, such as the U-Net and Inception-v3 networks, were used as a base architecture. We developed a better performing model that effectively segmented inner ears. Furthermore, an additional test was performed to prove the robustness of the system. By developing the INHEARIT-v2 system, expert interpretation became much easier and faster, and more objective analyses were possible.
The best performance, which was based on the mean area under the receiver operating characteristic curve from the fivefold cross-validation, was achieved by the 3intoUNet network, which was based on the U-Net architecture. The U-Net network is designed for medical image segmentation [
With regard to the performance of the system in automatic representative slice selection, when compared to physicians’ choices, the system’s choices had a gap distance of ≤2 for 100% (240/240) of the test data set. The physicians' goal was to select a key slice of an organ based on the organ's anatomical relationship with other organs [
Several concerns might arise when artificial intelligence systems are used in real clinical settings. In this study, we found that using such artificial intelligence systems could be an alternative to manually measuring hydrops ratios in real settings. However, this does not mean that Ménière disease can only be diagnosed by this system. The cutoff value for the EH ratio was not clearly defined during the diagnosis of definite Ménière disease. Future studies for estimating the EH ratio cutoff value in the diagnosis of definite Ménière disease are still needed. In addition, HYDROPS-Mi2 images were used to analyze EH ratios. However, these images were acquired via software modification, and hydropses in modified images can be more exaggerated compared to those in original images. The final concern is that Ménière disease is a multifactorial disease, which means that making an accurate diagnosis is typically a very complex process. Other diseases that mimic the symptoms of Ménière disease should be ruled out in clinical settings. The main purpose of hydrops measurement via artificial intelligence is to provide clinical support, which can be helpful for medical professionals when making a final clinical diagnosis.
Future studies can include additional normal control subjects to determine the optimal EH ratio threshold by comparing individuals with Ménière disease and healthy individuals without the disease. However, the association between clinical symptoms and EHs is not uniform from patient to patient; thus, such analyses require a clinician's comprehensive judgment [
Notably, we fully automated the calculation of EH ratios by developing an analytic system—the INHEARIT-v2 system—by using MRI and deep learning, which have important clinical implications. Although several aspects should be further investigated, this framework will be a helpful tool for clinicians who adopt an MRI analysis approach for diagnosing patients with Ménière disease.
We developed a fully automated system—the INHEARIT-v2 system—for calculating EH ratios by using deep learning and MRI. The proposed system can quickly and accurately analyze EHs without the intervention of an expert of various inner ear diseases, including Ménière disease experts. This automatic system can perform objective and time-saving analyses for assessing the EH ratios in the inner ear magnetic resonance images of patients with Ménière disease.
The concept of the 3intoUNet network.
endolymphatic hydrops
hybrid of the reversed image of the positive endolymph signal and native image of the positive perilymph signal
hybrid of the reversed image of the positive endolymph signal and native image of the positive perilymph signal image multiplied by T2-weighted magnetic resonance cisternography
intraclass correlation coefficient
inner-ear hydrops estimation via artificial intelligence
inner-ear hydrops estimation via artificial intelligence-version 2
intersection over union
magnetic resonance cisternography
magnetic resonance imaging
positive endolymph image
positive perilymph image
region of interest
Visual Geometry Group
This work was supported by National Research Foundation of Korea grants, which are funded by the Korean government (the Ministry of Science and ICT; grants NRF-2020R1F1A1073904 and NRF-2021M3E5D2A01022399), and the Future Medicine 20*30 Project of the Samsung Medical Center (grant SMX1210791).
BHC, WHC, and JWK designed the research. WHC and YSC collected the data. CJP, YKK, and KK analyzed the data. CJP and YSC wrote the main paper. BHC, WHC, and HJK provided critical revisions, discussed the results and implications, and commented on the manuscript at all stages.
None declared.