This study is conducted in Korea, as it has many medical facilities, and closures occur frequently. Therefore, using AI methods, Korea serves as a good experimental setting to observe and predict the closure of medical facilities. Institutionally, Korea has adopted the national health insurance system. For the same types of medical care, each medical facility receives an equal amount of reimbursement because of the setting of the national health insurance systems. There are also many private and public medical facilities, such as small hospitals, MCs, and DCs. Patients can choose any primary care clinic without constriction. Judging from these facts, the market is very competitive for survival, and dynamic factors may affect the closure of medical facilities. This study targeted MCs and DCs as study participants because there are many clinics and frequent occurrences of events such as clinic closures. Approximately 30,000 MCs and 25,000 DCs are operating as of December 31, 2022. Owing to the high competition, there are several closures.

Regarding the recruitment settings and procedures of study participants, this study used a research data set from the Health Insurance Review and Assessment Service (HIRA). The HIRA is a third-party administrator providing professional medical claim review and assessment services to Korea’s National Health Insurance Service. Every health care organization, such as hospitals and clinics, should register or report their openness and closure of facility to the government through a Unified Portal Reporting Healthcare Resource [31]. This portal is run by the HIRA, which collects data for administrative work. In detail, this study took data from the HIRA on clinics that operated and closed for 2 years from 2020 to 2021. Then, the participants for this study were selected from operating and closed clinics based on region at a ratio of 2:1. The detailed method will be explained in the Study Design section. The second data set was from an open public data set on residential and household information [32] run by the Ministry of Interior and Safety. This study collected data such as area code, local population, population growth, and number of households from the Ministry of Interior and Safety. These data were analyzed by linking them with the region code to which the clinics belong. The last one was from information on subway stations provided by the National Geographic Information Institute [33]. Using the clinic’s location information, we identified whether there was a subway station within a 1 km radius or not. National Geographic Information Institute provided this information.

This study adopted a cross-sectional study design for over a 2-year observation period. The unit of analysis was individual MCs and DCs. For this study, the closure status of MCs and DCs during the period 2020-2021 was identified. By observing 2 time points on December 31, this study identified whether the clinics closed or not and coded them as a group of clinics closed (1) or not (0). Using the closed group, we selected the participants for this study from closed and operating medical facilities with a ratio of 1:2 in MCs and DCs, respectively, because the proportion of closed facilities was too low, constituting less than 5% of the total study population. Training and testing participants were processed with a 4:1 ratio and 5-fold cross-validations for 4 machine learning algorithms: LR, SVM, random forest (RF), and Extreme Gradient Boost (XGB) methods.

The amount of data was sufficient, so the model learning the characteristics of each variable would be suitable. Therefore, we used stratified cross-validation to maintain the same ratio of running and closed clinics for each fold to prevent bias instead of using simple K-fold cross-validation. This study selected the LR and SVM methods because the former is the most classical model among the various machine learning models, and the latter is considered the fundamental model of deep learning and performs well in dealing with complicated models. This study also selected the RF and XGB methods because they are most frequently used for machine learning, with moderate modeling performance. Using these methods, this study predicted the probability of closure of MCs and DCs. After developing the prediction model, we evaluated the prediction results. The results were expressed through cross-validation. Figure 1 demonstrates the data processing flow or diagram of this study.

This study predicted the closure or nonclosure of MCs and DCs, the major outcome variable. The other input variables were chosen based on previous studies but within the boundary of data availability: years of operation, number of patients, annual variation in the number of patients, having beds or not (for clinics), having computed tomography devices or not (for clinics), having magnetic resonance imaging devices or not (for clinics), number of physicians (for clinics) or dentists (for DCs), percentage of medical or dental specialists, percentage of nurses or hygienists, the local population of the year 2021, population growth rate (%), number of clinics or DCs within the local area, and having a subway station or not within 1 km.

The input variables were selected based on several previous studies. These studies used general statistical methods such as LR rather than machine learning. A study found that the closure of MCs was associated with years of operation, for-profit status, the proportion of specialists among all physicians, the number of clinics within the local area, the closure of DCs related to the proportion of specialized dentists among all dentists, the proportion of nurse and dental hygienists among all nursing staff, and several DCs within the local area [11]. Several studies on the closure of hospitals and business firms presented that closure was associated with factors such as facility age, financial status, affiliation status, and market factors [22,24,26,27]. This study used all available variables as input variables.

The annual number of patients was observed in 2020 for operating MCs and DCs. For closed MCs and DCs, the number of patients a year before closure was used. For the annual variation in the number of patients, the SD of the number of patients for 5 years was calculated. If clinics had less than 5 years of operation, we calculated the SD of the number of patients for the years of operation. The percentage of medical or dental specialists was calculated by (specialized physicians for clinics (dentists for DCs)/sum of physicians of clinics (dentists for DCs) × 100), and the percentage of nurses for MCs was calculated by (number of nurses/(number of nurses + nurse aids)× 100). The percentage of nurses for DCs was calculated by ((number of nurses + number of dental hygiene)/(number of nurses + number of dental hygiene + nurse aids)) × 100. Local area means an administrative district in Korea.

Various machine learning techniques are used to predict patients’ clinical behavior and decision-making [34,35]. This study used similar methods to those in previous studies. Before the main analysis, we took the following data preprocessing steps. First, by using the SAS program, we excluded all missing data. Second, as mentioned above, this study selected study participants from closed and running medical facilities with a ratio of 1:2 because the proportion of event groups, meaning closed facilities, was too low, being less than 5%. This process was conducted in MCs and DCs, respectively. This process was conducted with the SAS program, and this study used the propensity matching score. Third, for the cases of numeric valued data, this study rescaled them to have a normal distribution with a mean of 0 (SD 0) and a variance of 1, which is called the standard scaling process.

This study used MedCalc (version 14.8.1; MedCalc Software Ltd) to reanalyze the predicted results regarding the machine learning module. When we calculated the receiver operating characteristic curve comparison, Stata (version 11; StataCorp LLC) was used. Regarding the machine learning algorithm, this study used the LR, SVM, and RF, which are provided by the library of scikit-learn (version 0.23.2; Python Software Foundation) and xgboost (version 1.4.0; Python Software Foundation). Regarding feature selection, this study used an algorithm that shows the permutation importance, which the library of Eli5 provides for computing feature importance. This study used 23 and 19 variables to predict the closure of MCs and DCs, respectively. Key variables were extracted using the permutation importance from those variables. The sequential feature selection technique was also used. Finally, this study selected 5 and 6 variables of MCs and DCs, respectively, for model learning.

Regarding the validation methods of the models, this study used sensitivity and specificity indices following the definition of a previous study [36]. Regarding the prediction capacity of the models, this study used area under the curve (AUC), which was calculated based on sensitivity and specificity. According to the Standards for Reporting Diagnostic Accuracy Studies 2015 [37], it is suggested that specificity, sensitivity, and receiver operating characteristic curves be used for diagnostic evaluation. Thus, this study chose them and presented its study outputs. This study used Python (version 3.7.9; Python Software Foundation) to solve the main analysis.

Regarding AI methods and paper writing, this study discloses that no generative AI was used at any stage of writing this paper. Finally, the computer operating and hardware system were Intel Core i9-10900 CPU @ 2.80 GHz and Windows 10 OS.

This study was approved by the institutional review board (IRB) of the HIRA on February 10, 2022 in Korea (2022-010-001). The IRB letter says that it is exempted from proceeding with the formal IRB review due to the use of the secondary health insurance administrative data. The letter also says that it has nothing to do with obtaining informed consent from research participants because the study participants are individual clinics.

Table 1 presents the basic characteristics of clinics and DCs regarding closure status. Closed MCs are more likely to have longer operating years, fewer patients, and less variation in the number of annual patients compared to those operating. Closed DCs are less likely to be specialized and have fewer specialized nurses compared to the others.

Figure 2 presents the model performance results of 4 models in MCs and DCs. In MCs, model performance was highest in SVM for the AUC index, followed by RF, XGB, and LR. In DCs, model performance was highest in XGB, followed by RF, LR, and SVM.

Table 2 presents the performance indicator for each model. In the cross-validation, the AUC of MCs was highest in SVM (0.762), followed by RF (0.736). The AUC of DCs was highest in XGB (0.700), followed by RF (0.687).

Years of operation, population growth, number of residents within the local area, and percentage of medical specialists are crucial factors for closing MCs. In contrast, number of patients, annual variation in the number of patients, years of operation, percentage of dental specialists, and number of residents within the local area are considered essential factors affecting DC closures.

Table 3 presents the crucial factors measured by permutation importance across 4 models. The year of operation for MCs had the highest score (0.194), followed by population growth (0.110). For DCs, the number of patients had the highest score (0.096), followed by annual variation in the number of patients (0.060).

This study found that it is feasible to predict the closure of MCs and DCs using machine learning methods with a fair level of the validation score measured by the AUC. Interestingly, MCs and DCs have different best-fitting models. The best-fitting MC and DC closure models were in SVM and XGB, respectively. Finally, except for years of operation, different important features affected the closure of MCs and DCs. Years of operation were an important feature affecting the closure of both MCs and DCs. External environmental factors, such as population growth and local residential population, largely influenced the closure of clinics. However, the closure of DCs was influenced by the internal characteristics of the facility, such as the number of patients and annual variation in the number of patients.

This study indicates the feasibility of predicting clinic closure with machine learning methods. This result aligns with previous studies in which many AI methods well predicted patients’ clinical behaviors [34,35] and partially predicted hospital closure and general firm bankruptcy [12-14,24]. Other than that, we could not find any similar previous studies targeting MCs and DCs; thus, it was impossible to compare the results of this study with others. However, our study results suggest that the same machine learning technologies could be applied to predicting the closure of medical facilities, such as many long-term care facilities, pharmacies, and clinics with oriental medicine.

Regarding the model performance to predict closure, SVM and XGB were the best-fitting models to predict the closure of MCs and DCs, respectively. A very interesting finding is that each study group has its best-fitting model. While a study on bankruptcy prediction presented that XGB is the best performer, the best-fit model for predicting the annual number of inpatients and outpatients was RF and linear regressor, respectively [25,38]. These results suggest that future research should refer to the optimal models of previous studies and include such models as alternatives in their research methods.

For cross-validation using the AUC score, MCs had the highest score in SVM (0.762) followed by RF (0.736) and DCs had the highest score in XGB (0.700) followed by RF (0.687). These study results suggest 2 things. The first is about the models’ cross-validation per se. Generally, an AUC score above 0.70 would be considered “fair” for validating the model [39]. Several previous studies dealing with various health care participants have a broad range of AUC and modeling accuracy ranging from 0.69 [24] to 0.85 AUC [40]. Compared to these previous validation scores, this study has a fair cross-validation score, and thus, there is ample potential to develop further good models based on this study. Second, this study suggests that the model performance of MCs and DCs needs to improve further. As evidenced by the results of this study, the overall cross-validation score of each model in MCs is slightly higher than that of DCs. Further search and analysis are necessary to investigate why these are different. This study suggests that methodological approaches that involve selecting different input variables by each model with their own are necessary.

This study found that years of operation were an important factor affecting the closure of MCs and DCs. Clinics performing medical practices over a longer period in 1 place might consider moving to other places, which is why factors such as years of operation are significant factors affecting clinic closure. Our study results showed that external environmental factors, such as population growth and local residential population, are essential to consider in the closure of clinics, and the internal characteristics of DCs, such as the number of patients and annual variation in the number of patients, critically affect the closure of DCs. The results of this study partially align with those of a previous study in which factors such as medical and nurse specialties (both in MCs and DCs), year of operation (in both), and market competition are closely associated with the closure of medical facilities [11]. Nevertheless, there are some differences in that while the previous study was based only on LR investigating factors associated with the closure of MCs, this study involved predicting and identifying critical factors affecting the closure of clinics with various AI methods.

These research results tell people who want to open a new MC or DC what factors are related to business closure. This study argues that these factors must be considered in countries where many MCs and DCs are operating. This study used several machine learning methods, but they needed more accuracy. This study presents the challenge of increasing the accuracy of the methodology, thus providing fundamental reference values for these analysis methods.

Regarding the results of this study concerning hospital closure, there are distinctive differences between hospitals and clinics. While factors such as affiliation status, financial distress, size of the hospital, locality, low-profit margin, and market competition were critical factors affecting hospital closure [22,26,27,29,30], features such as years of operation, local population, and number of patients were significant factors affecting MC and DC closure [11]. However, direct comparisons between hospitals and clinics are still unreasonable because the participants of this study are different.

This study is valuable in several respects. First, this study is the first to predict the closure of MCs and DCs. There have been several studies on the closure of hospitals but not clinics. Second, studies on clinic closure are very important because clinics play a dominant role in providing health care to the community residents. Identifying closure factors is crucial to prevent the breakdown of the health care delivery systems of local communities. By investigating their survival through this kind of study, we can obtain useful information to increase the sustainability of their business survival. Third, this study had highly valid data sets comprising national data. A study using a large cumulative data set would be invaluable because the validity of study results increases as the sample size of a study increases.

This study has several limitations. First, although this study was based on a 2-year national data set, few clinics have closed. This study conducted predictions using 2-year aggregated data by selecting operating clinics and matching them with closed clinics. This process might lead to the loss of some valuable information. However, this study had a rigorous methodology in that it increased comparability by equally weighting the risk factors to those study participants within the same locality. Second, although financial factors are critical factors affecting hospital closure [22,27], this study did not include that variable. However, to minimize its impact, this study included the number of patients as a potential proxy variable of financial factors. Third, this study design is of closed clinics, and running clinics were selected at a ratio of 1:2 based on the locality of study participants using the propensity matching score of LR, which might result in an under-sampling issue and, thus, would have the risk of overfitting. We suggest that future research could consider a hypothesis-free approach, such as repeated undersampling, to overcome this study’s limitations. Fourth, the data collection period is within the COVID-19 pandemic. The COVID-19 pandemic is very special, in which the closure of small clinics might be common. Small health care institutions are more greatly affected by external forces [7]. The data from the pandemic period would differ from those we see today. Thus, the interpretation of study results needs consideration of this background. However, although data were collected during this unique period, this study argues that the impact of COVID-19 would equally affect those clinics running and closed. This may minimize its negative impact on study data. Finally, the results of this study may be limited to Korea and countries with health care delivery systems similar to those in Korea. Suppose the operation or closure of MCs is dependent on governmental plans. In that case, the results of this study may not apply to those countries because this study is based on free business medical markets.

This study predicted the closure status of MCs and DCs and found that composing models that predict the closure status of clinics is feasible. Among the 4 models, SVM was the best-fit model for MCs and XGB for DCs. Years of operation were an important factor that critically affected the closure of MCs and DCs. Additionally, environmental factors critically affected the closure of MCs; in contrast, DC’s internal features, such as the number of patients, were critical factors affecting the closure of DCs. These discoveries increase the range of our knowledge and understanding that machine learning techniques could be useful in predicting the closure of small medical institutions and applicable to long-term care hospitals, nursing homes, and pharmacies. This study also indirectly suggests that governments could conserve health care resources and prevent wastage if they develop good prediction models and provide information to medical or potential medical providers. We hope this study will inspire similar studies with further dynamic methods, leading to knowledge expansion and providing various insights to policy makers and researchers in related fields.