The PSG data were collected from Samsung Medical Center (SMC, n=4257) and Ewha Womans University Medical Center Seoul Hospital (EUMCSH, n=365). We randomly divided the SMC data set into training and testing sets in a ratio of 8:2. In this paper, the training set (testing set) is referred to as the SMC training set (SMC testing set). We applied 3 XGBoost models to the SMC training set, classifying participants with OSA, COMISA, and insomnia, respectively, using items from sleep-related questionnaires and demographic characteristics as input features (Figure 1). To calculate the feature importance, the Shapley additive explanations (SHAP) [38] method is used. Based on computed feature importance, features with high importance were selected. Then, we made new XGBoost models predicting sleep disorders using only these selected features. The classification performance of the model was verified using the SMC testing set. The EUMCSH data set was used as an independent test set to check the universality of the model.

In this work, we used XGBoost for classification. XGBoost is a tree-based machine learning algorithm using gradient boosting [36] and has been widely used in medical diagnosis because of its high accuracy and performance, straightforward interpretation, and ability to handle missing values [43-47]. Logistic regression (LR), random forest (RF), and support vector classifier were also implemented for comparison.

The SMC data set was randomly divided into training and testing sets in a ratio of 8:2 to develop a model. To deal with the imbalanced data, we used stratified splitting to maintain the ratio of positive values for the training and testing sets. We used stratified 5-fold cross-validation to train the XGBoost model, dividing the training set into 5-folds and training the model using 4-folds while testing on the remaining fold. We used Bayesian optimization to choose hyperparameters that maximize the average target function of the 5-fold cross-validation [48]. The XGBoost model was tuned based on max_depth, learning_rate, n_estimators, gamma, min_child_weight, max_delta_step, subsample, and colsample_bytree, and the ranges of the values were 2-10, 0.001-0.3, 50-4000, 0.01-1, 1-10, 0-0.1, 0.7-0.8, and 0.5-0.99, respectively. For OSA, the AUROC value was used as the target function of the model. For COMISA and insomnia, since the labels were imbalanced, we chose the F₁ score as a target function, which tends to perform better than the AUROC for imbalanced data [49]. In addition, we gave different weights to positive and negative classes (proportional to the inverse of the class size) to prevent the model from ignoring the minority class during training.

To identify important features, we built the XGBoost model using the SMC data set and the SHAP method [38]. SHAP is a game theoretic method that explains the prediction of the model by calculating the impact of each feature on the model output. Positive (negative) SHAP values indicate a positive (negative) effect on the model’s output. Feature importance can be calculated by averaging the absolute SHAP values for all data.

We used AUROC and the area under the precision-recall curve (AUPRC) to evaluate the performance of the classification model. The receiver operating characteristics (ROC) curve plots the false-positive rate (the fraction of nonpatients mispredicted as a patient among all nonpatients) and true-positive rate (the fraction of correctly inferred patients among all predicted patients) as the probability threshold varies. The AUROC value (a value between 0 and 1) indicates the probability that the prediction model will successfully assign a higher risk value to a random positive participant than a random negative participant. When the data are imbalanced, AUPRC has been used to quantify the model performance [50]. A precision-recall curve plots precision (the fraction of correctly inferred patients among all predicted patients) and recall (true positive rate) as the probability threshold varies. Since the labels for COMISA and insomnia are imbalanced, we also use the AUPRC value to evaluate the performance of our model. We also provide the 95% CIs of the AUROC and AUPRC values on both training sets, which were estimated using bootstrapping (n=10,000). XGBoost and other classification models were implemented in Python (version 3.8) using Scikit-Learn [51] and XGBoost [36] packages. Bayesian optimization was implemented using BayesianOptimization packages [52].

The website was built using React JS (Meta Open Source) for the front end and Django (Django Software Foundation) for the back end. React JS provides an intuitive and easy-to-use interface through rich JS libraries, while Django allows for easy connection of the model with the back-end server using Python. The web app connects with the model through the Django server, which loads it directly from a checkpoint. The website is designed to be easy to use and allows users to predict their risk of having OSA, COMISA, and insomnia and to see how that risk changes with changes in weight or age. It includes (1) questionnaires for the user to fill out, (2) a prediction of the risk of having a sleep disorder, (3) a visualization of how risk changes with changes in weight or age, and (4) a description of each sleep disorder. Since XGBoost is a stochastic model, the results may vary with different random seeds. To make a robust prediction, we integrated the output from 10 models by changing the random seed when providing a change in risk with varying weight or age. Using 10 different results, we provided the upper and lower CIs as well as the mean prediction.

The demographic and clinical characteristics of the SMC data set are listed in Table 1. The average age was 50.5 (SD 14.1) years, and the average BMI was 25.8 (SD 4.2) kg/m². According to diagnosis, the average age of the COMISA group was 55.0 (SD 12.9) years; for the OSA group it was 52.1 (SD 13.0) years, the insomnia group 50.0 (SD 13.8) years, and the negative group 43.0 (SD 14.9) years. The proportion of males was the highest in the OSA group at 86% (1770/2059), followed by the COMISA group at 77.1% (623/808), the negative group at 62.1% (577/929), and the insomnia group at 41.4% (191/461).

The demographic and clinical characteristics of the EUMCSH data set are listed in Table 2. The average age was 57.6 (SD 14.7) years and the average BMI was 26.4 (SD 4.6) kg/m². According to diagnosis, the average age of the OSA group was 59.3 (SD 14.1) years; for the COMISA group it was 58.7 (SD 13.3) years, the insomnia group 49.7 (SD 16.2) years, and the negative group 49.2 (SD 16.5) years. The proportion of males was highest in the OSA group at 70.5% (146/207), followed by the COMISA group at 55% (55/100), the negative group at 40% (14/35), and the insomnia group at 39.1% (9/23).

We aim to develop an algorithm that can accurately predict the risk of 3 sleep disorders—OSA, COMISA, and insomnia—using a simple set of questions selected from 22 items from sleep-related questionnaires and 8 demographic characteristics. To achieve this, we first built 3 XGBoost models using 30 input features to predict participants’ risk of having OSA, COMISA, and insomnia with the SMC training set. We then calculated the importance of these 30 features using the SHAP method. To identify the key features for predicting all 3 disorders simultaneously, we summed the feature importance for each of the sleep disorders (Figure 2). Then, we selected the top 5 items from the sleep-related questionnaires (ISI1b, ISI1a, ISI5, ISI1c, and ISI2) whose feature importance is greater than 1.5. Using the same threshold, age was selected among the demographic characteristics. Additionally, we selected 3 other characteristics that can be easily answered (weight, BMI, and sex) from a group of 6 characteristics (sex, weight, BMI, neck circumference, waist circumference, and hip circumference) with feature importance values close to 1. In total, we used 9 features (5 items from the sleep-related questionnaires and 4 demographic characteristics) to develop a simple questionnaire predicting sleep disorders (SLEEPS) based on 3 XGBoost models trained using the SMC training set to predict the risk of OSA, COMISA, and insomnia.

We tested the performance of SLEEPS using the SMC testing set (Table 3 and Figure 3A). SLEEPS has high accuracy in predicting the risk of OSA (AUROC=0.897), COMISA (AUROC=0.947), and insomnia (AUROC=0.922). The AUPRC values for each disorder are as follows: OSA (0.877), COMISA (0.786), and insomnia (0.611). Note that SLEEPS outperformed the recently developed OSA screening model BASH-GN [29] (AUROC=0.78 and AUPRC=0.76), although BASH-GN outperformed the commonly used OSA screening questionnaires such as STOP-BANG [25] and Berlin [26].

The AUROC values for all 3 sleep disorders are greater than 0.897, indicating that SLEEPS can accurately predict the risk of these disorders (Figure 3A). The seemingly low AUPRC values for COMISA and insomnia are due to the imbalance in the data, with a higher proportion of negative cases. To account for this, we compared the AUPRC values to the AUPRC baseline, which is the proportion of positive cases in the data. The OSA label is balanced, with a similar proportion of positive and negative cases, and SLEEPS has a high AUPRC value for predicting OSA (0.877). Despite the imbalanced labels for insomnia and COMISA (AUPRC baselines of 0.108 and 0.190, respectively), SLEEPS still has high AUPRC values (0.611 and 0.786, respectively), demonstrating its ability to distinguish between those with and without these disorders even with imbalanced data.

To evaluate the generalizability of SLEEPS, we also tested it using the EUMCSH data set as an independent test set. SLEEPS maintained similar performance on this new data set (Figure 3B and Table 3), showing that it is scalable and not dependent on specific data. In particular, the AUPRC values for OSA and COMISA increased for the EUMCSH data set, while the AUPRC value for insomnia decreased due to the more imbalanced data in the EUMCSH data set (AUPRC baseline=0.063). To investigate the low AUPRC value in predicting insomnia within the EUMCSH data set, we examined the model’s performance using a balanced subset of the data. We randomly sampled non-insomnia participants from the EUMCSH data set to match the number of insomnia participants, thus creating a perfectly balanced data set. Subsequently, we calculated the AUPRC value of SLEEPS on this balanced data. This process was repeated 10,000 times, and the results consistently showed higher AUPRC values compared to using the entire EUMCSH data set (Table S1 in Multimedia Appendix 1). This suggests that the low AUPRC value for predicting insomnia in the EUMCSH data set is primarily attributable to the high data imbalance. Furthermore, the SLEEPS model demonstrates high performance when applied to a balanced data set.

In contrast to previous OSA screening studies [25,26,30,31,33,35], we separated COMISA from OSA. To investigate the importance of this separation, we merged COMISA with OSA to create a new OSA label and retrained the XGBoost model predicting this label using the SMC training set. We then evaluated the model’s performance on the SMC testing set and EUMCSH data set (Table S2 in Multimedia Appendix 1). Interestingly, for both testing sets, the AUROC and AUPRC values of the model decreased for merged OSA, emphasizing that distinguishing COMISA from OSA is critical for accurate prediction. On the other hand, when we merged COMISA with insomnia and retrained the model, the prediction performance for insomnia increased for both testing sets, indicating that it is challenging to distinguish COMISA from insomnia.

We also examined if the performance of SLEEPS could be improved by adding additional features. We had not previously included neck, waist, and hip circumferences as input features because they are not easily measurable (Figure 2 inset). Thus, we created a new XGBoost model that considers these 3 features in addition to the 9 previously selected features. Even after adding the 3 input features, there is no significant improvement in prediction (Table S3 in Multimedia Appendix 1), indicating that the addition of these features did not significantly improve the model.

Furthermore, we also verified that XGBoost was the best choice for our classification model by developing and testing 3 other popular models: LR, RF, and support vector classifier. XGBoost outperformed the other models for all 3 sleep disorders on both the SMC testing set and the EUMCSH data set (Figures S1 and S2 and Tables S3 and S4 in Multimedia Appendix 1). In particular, the AUPRC for COMISA and insomnia was relatively higher for XGBoost than for the other 3 models, indicating that SLEEPS can accurately predict the risk even when the data are imbalanced.

To understand how SLEEPS differentiates between individuals with sleep disorders and those without, we calculated the SHAP values of the 9 features used in the model. A positive (negative) SHAP value indicates that the feature has a positive (negative) impact on the model’s output. We visualized the distribution of SHAP values for each feature using a beeswarm plot, where each dot corresponds to one participant and its color represents the SHAP value, with blue corresponding to low values and red corresponding to high values (Figure 4). For example, males (encoded as 0, blue dots) tend to have a higher chance of having OSA and COMISA than females (encoded as 1, red dots; Figure 4A and B). However, females have higher chances of having insomnia than men (Figure 4C). This is consistent with a previous study showing that females generally report having a greater sleep need and more frequently report poor or insufficient sleep than males [53]. A meta-analysis study also reported that the prevalence of insomnia is approximately 1.41 times more common in females than males [54]. Overall, the demographic characteristics (age, sex, weight, and BMI) have similar effects on OSA and COMISA, but opposite effects on insomnia. On the other hand, the questionnaire items (ISI1a, ISI1b, ISI1c, ISI2, and ISI5) have similar effects on COMISA and insomnia, but opposite effects on OSA (Figure 4). The beeswarm plot for the EUMCSH data set (Figure S3 in Multimedia Appendix 1) also shows a consistent tendency for each feature.

In the case of COMISA and insomnia, the labels in the data are imbalanced. Therefore, it might be misleading to interpret the model output (a number between 0 and 1) directly as the “probability” or “risk” of having a sleep disorder. To verify this issue, we divided the range of model output [0,1] into 10 equal intervals and observed the distribution of model output values for participants with and without a sleep disorder, respectively (Figure S4 in Multimedia Appendix 1). For each interval, we calculated the percentage of participants with model output in that interval who had sleep disorders (Figure 5). We then used linear regression to approximate this distribution (black line in Figure 5). Surprisingly, the linear fitting has a high R² value in all 3 cases. Since we can consider the entire SMC data set as the representable data of the “ground truth” (or likelihood) of Korean individuals, the y-value for specific model output can be interpreted as the “probability” of having a particular sleep disorder using the Bayes formula. Furthermore, all 3 regression lines are similar to y=x, indicating that the model output can be interpreted as the probability of having a particular sleep disorder.

SLEEPS allows us to calculate the probability of having each of the 3 sleep disorders. In addition to this, we can also provide specific guidelines related to the 3 sleep disorders. Out of the 4 demographic characteristics used in SLEEPS, weight can be easily changed, and age increases over time. By varying the age or weight while holding the values of other features constant, we can predict how the probability of having the 3 sleep disorders changes as age or weight varies. This can provide quantitative answers to questions such as “How much weight should I lose to reduce my risk of having OSA below 0.5?” or “How will my risk of having COMISA change as I age?”

To make SLEEPS more widely available, we have also created a publicly available website [37] that predicts the risk of the 3 sleep disorders based on SLEEPS (Figure 6). When users input their answers to the 9 questions (Figure 6A), the website shows the predicted risk of the user having each of the 3 sleep disorders (Figure 6B). The website also provides a graph showing the predicted change in risk as weight varies or age increases (Figure 6C), along with the CIs for the prediction.