Published on in Vol 26 (2024)

Preprints (earlier versions) of this paper are available at, first published .
Machine Learning–Based Prediction of Suicidal Thinking in Adolescents by Derivation and Validation in 3 Independent Worldwide Cohorts: Algorithm Development and Validation Study

Machine Learning–Based Prediction of Suicidal Thinking in Adolescents by Derivation and Validation in 3 Independent Worldwide Cohorts: Algorithm Development and Validation Study

Machine Learning–Based Prediction of Suicidal Thinking in Adolescents by Derivation and Validation in 3 Independent Worldwide Cohorts: Algorithm Development and Validation Study

Original Paper

1Center for Digital Health, Medical Science Research Institute, Kyung Hee University College of Medicine, Seoul, Republic of Korea

2Department of Applied Information Engineering, Yonsei University, Seoul, Republic of Korea

3Department of Precision Medicine, Kyung Hee University College of Medicine, Seoul, Republic of Korea

4Department of Regulatory Science, Kyung Hee University, Seoul, Republic of Korea

5Division of Sleep Medicine, Harvard Medical School, Boston, MA, United States

6Department of Anesthesia, Critical Care and Pain Medicine, Massachusetts General Hospital, Boston, MA, United States

7Department of Business Administration, Kyung Hee University School of Management, Seoul, Republic of Korea

8Department of Korean Medicine, Kyung Hee University College of Korean Medicine, Seoul, Republic of Korea

9Assistance Publique–Hôpitaux de Marseille (APHM), CEReSS–Health Service Research and Quality of Life Center, Aix–Marseille University, Marseille, France

10Department of Pediatrics, Kyung Hee University Medical Center, Kyung Hee University College of Medicine, Seoul, Republic of Korea

*these authors contributed equally

Corresponding Author:

Dong Keon Yon, MD, PhD

Center for Digital Health

Medical Science Research Institute

Kyung Hee University College of Medicine

23 Kyungheedae–ro


Seoul, 02447

Republic of Korea

Phone: 82 2 6935 2476

Fax:82 504 478 0201


Background: Suicide is the second-leading cause of death among adolescents and is associated with clusters of suicides. Despite numerous studies on this preventable cause of death, the focus has primarily been on single nations and traditional statistical methods.

Objective: This study aims to develop a predictive model for adolescent suicidal thinking using multinational data sets and machine learning (ML).

Methods: We used data from the Korea Youth Risk Behavior Web-based Survey with 566,875 adolescents aged between 13 and 18 years and conducted external validation using the Youth Risk Behavior Survey with 103,874 adolescents and Norway’s University National General Survey with 19,574 adolescents. Several tree-based ML models were developed, and feature importance and Shapley additive explanations values were analyzed to identify risk factors for adolescent suicidal thinking.

Results: When trained on the Korea Youth Risk Behavior Web-based Survey data from South Korea with a 95% CI, the XGBoost model reported an area under the receiver operating characteristic (AUROC) curve of 90.06% (95% CI 89.97-90.16), displaying superior performance compared to other models. For external validation using the Youth Risk Behavior Survey data from the United States and the University National General Survey from Norway, the XGBoost model achieved AUROCs of 83.09% and 81.27%, respectively. Across all data sets, XGBoost consistently outperformed the other models with the highest AUROC score, and was selected as the optimal model. In terms of predictors of suicidal thinking, feelings of sadness and despair were the most influential, accounting for 57.4% of the impact, followed by stress status at 19.8%. This was followed by age (5.7%), household income (4%), academic achievement (3.4%), sex (2.1%), and others, which contributed less than 2% each.

Conclusions: This study used ML by integrating diverse data sets from 3 countries to address adolescent suicide. The findings highlight the important role of emotional health indicators in predicting suicidal thinking among adolescents. Specifically, sadness and despair were identified as the most significant predictors, followed by stressful conditions and age. These findings emphasize the critical need for early diagnosis and prevention of mental health issues during adolescence.

J Med Internet Res 2024;26:e55913



Adolescent suicide stands out as a prominent global public health concern, with its rank as the second leading cause of death among young populations underscoring its severity [1,2] Notably, adolescence is a phase characterized by an amplified suicide risk [3]. Concerningly, some geographic regions are experiencing a surge in suicide clusters, where the instances of suicide exceed the typical levels [4]. Research into these clusters indicates that individuals younger than 25 years are up to 4 times more likely to be affected by suicide [5]. Since suicide is preventable in the early stages, there is a pressing need for action through rigorous mental health strategies and proactive educational interventions [6].

While various methodologies have been proposed to prevent suicidal thinking in adolescents, many lack empirical outcomes and often fail to identify key determinants [7-9]. A significant gap remains in accurately assessing the risk of suicidal thinking for individual adolescents [2,10,11]. Recent advances in machine learning (ML) methodologies have shown promise in addressing the challenges of adolescent suicidal tendencies. Studies leveraging boosted ML [12], daily data analysis through classification and regression trees [13], and risk and protective factor frameworks [14] have begun to unpack the complex interplay of factors contributing to suicidal thinking among adolescents. However, these studies have also highlighted limitations, including a focus on specific socioeconomic or short-term predictors and a lack of comprehensive risk profiles integrating emotional, social, and psychological variables [12-14].

Therefore, in this study, we developed a predictive model for suicidal thinking among adolescents, using advanced ML algorithms. Addressing the gaps identified in earlier research, our model incorporates a broader array of factors, including family dynamics, emotional well-being, academic performance, and general health indicators. Across distinct adolescent cohorts from South Korea, Norway, and the United States, we aimed for a comprehensive multinational approach. By refining our approach based on previous studies’ insights, this research aims to highlight the preventability of suicide and influence mental health clinicians and policy makers to develop more effective preventive measures and supportive programs.

Study Design and Participants

This study aimed to develop an ML model to predict suicidal thinking among Korean adolescents. Our approach used multiple variables extracted from 3 distinct, large-scale international data sources: the Korea Youth Risk Behavior Web-based Survey (KYRBS) [15,16], the Youth Risk Behavior Survey (YRBS), and Norway’s nationwide University National General Survey (Ungdata).

Data Preparation and Harmonization

Initial data preprocessing involved adjusting the sample sizes after the removal of missing values: KYRBS from 1,145,178 to 566,875, YRBS from 438,566 to 103,874, and Ungdata from 89,077 to 19,574. We analyzed data from adolescents aged between 13 and 18 years who participated in the KYRBS from 2009 to 2021, the YRBS in 2021, and Ungdata from 2017 to 2019. The primary outcome, termed “current suicidal thinking,” was derived from participants’ affirmative responses to the question, “During the past 12 months, did you ever seriously consider attempting suicide?” This outcome indicated that participants had contemplated serious suicidal thinking at least once in the preceding year. The analysis considered several covariates: region, age, sex, BMI (kg/m2), academic achievement, household income, smoking status, alcoholic consumption, stress status, feelings of sadness and despair, exercise habits, and screen time (Figure S1 in Multimedia Appendix 1) [17].

We harmonized the data sets for XGBoost model compatibility, addressing the challenge posed by different variable configurations within the same questions. Our preprocessing aligned each variable across the KYRBS, YRBS, and Ungdata data sets, ensuring they matched in terms of content and format. Recognizing the potential disparities in variable configurations across these data sets, we standardized the variable names, formats, and scales, focusing on key features such as demographic information, behavioral factors, psychosocial aspects, and environmental influences that could serve as predictors for suicidal thinking. To ensure consistency, we adopted the following strategic approach to cases where YRBS and Ungdata were missing certain features: by calculating the median of the missing variables in the KYRBS data set, we were able to effectively impute the missing values to maintain the integrity and comparability of the data set compilations. Through variable alignment and addressing missing data, we successfully harnessed the diverse strengths of each data set, facilitating the development of a comprehensive model designed to address adolescent suicidal thinking effectively.

ML Model Development

Our ML model underwent training and validation to ensure its predictive accuracy in identifying suicidal thinking. We used the KYRBS data set to build a model tailored to predict suicidal thinking among Korean adolescents aged between 13 and 18 years. Recognizing the intricate characteristics of the data, we used a variety of tree-based ML techniques, including XGBoost, adaptive boosting (AdaBoost), light gradient-boosting machine (LightGBM), and random forest to train the data set for our modeling process [18]. Before this, data preprocessing measures, such as addressing missing values and encoding categorical variables, were executed to maintain data integrity and optimize the data for the modeling phase.

We adopted the 10-fold cross-validation method, dividing the initial data set into 10 equal-sized subsets, to rigorously assess the performance efficacy of the ML model. Of these, 9 are designated for model training, while the remaining subset serves as validation [19]. The process iterates 10 times, ensuring each subset undergoes validation at least once. During each cycle, we computed various performance metrics such as area under the receiver operating characteristic (AUROC) curve, sensitivity, specificity, accuracy, and balanced accuracy, along with their respective 95% CIs [20-25]. The 95% CIs provide a range of possible values for the model’s performance metrics, allowing us to assess the stability and generalizability of the model. For a visual representation of the model efficacy, we used visualization methods, primarily the receiver operating characteristic curve. After 10 iterations, the metrics from each were averaged to determine the final performance evaluation.

We trained our model on the KYRBS data set using 10-fold cross-validation. The model trained on the KYRBS data set was then externally validated with YRBS and Ungdata data sets preprocessed with the same column structure as KYRBS. This rigorous process reinforced the reliability of our model’s performance trained on the KYRBS data set [26]. Among the 4 models tested, XGBoost consistently yielded the highest AUROC scores across all data sets, leading to its selection as the primary model.

We performed hyperparameter tuning using GridSearchCV to optimize the performance of the XGBoost model, prioritizing the maximization of the AUROC score to determine the optimal hyperparameter combination. Hyperparameters were carefully selected for improved performance: the gbtree booster was used for its effectiveness in classification tasks, and the logloss evaluation metric was chosen to ensure accurate probability estimations. We set the learning rate at 0.08 to balance training speed with model accuracy, and the max depth was capped at 5 to prevent overfitting while allowing the model to capture complex patterns. Additionally, 350 trees (n_estimators) were used to construct a robust model, with further adjustments made to parameters like “scale_pos_weight” and subsample to address class imbalance and enhance model stability. These adjustments were important in refining our model’s predictive capabilities and are detailed in Table S1 in Multimedia Appendix 1. In order to interpret and gain insights into the model predictions, we used Shapley additive explanations (SHAP) values, a unified measure derived from cooperative game theory. Data set variables were analyzed with SAS software (version 9.3; SAS Institute Inc), and ML analysis was performed using Python (version 3.11.4; Python Software Foundation). The main Python libraries used are as follows: NumPy (version 1.26.0; Python Software Foundation) for data arrays and operations, and Pandas (version 2.1.0; Python Software Foundation) for data manipulation and analysis. All 3—scikit-learn (version 1.2.2; scikit-learn development team), TensorFlow-gpu (version 2.6.0; Tensor development team), and Keras (version 2.6.0; Keras development team)—were used for constructing and training ML models [27]. Additionally, the SHAP package (version 0.42.1) was used to interpret the ML models and for its explanation capabilities [28].

Software and Libraries

All computations, model training, and evaluations were executed using Python (version 3.11.4). Key libraries from our toolbox included scikit-learn (version 1.2.2), NumPy (version 1.24.0), and Pandas (version 2.1.0) for ML tasks and data wrangling. Visualization was facilitated using Matplotlib (version 3.7.2) and Seaborn (version 0.12.2).

Ethical Considerations

The study protocol was approved by the institutional review board of the Korean Disease Control and Prevention Agency (2014-06EXP-02-P-A), the US Centers for Disease Control and Prevention (#1969.0), the Norwegian Centre for Research Data and Data Protection Office of Inland Hospital Trust (18778329) and by the local law of the Population Health Promotion Act 19 (117058) form of the Korean government. All participants provided written informed consent. This research followed the guidelines outlined in the TRIPOD (Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis) statement (Table S2 in Multimedia Appendix 1).

Demographic Characteristics

This study was conducted to develop a ML-based predictive model for suicidal thinking among adolescents aged between 13 and 18 years. After collecting independent data from 3 countries, covariates were standardized for the ML prediction modeling process (Figures 1 and 2).

Figure 1. Study architecture. AdaBoost: adaptive boosting; KYRBS: Korea Youth Risk Behavior Web-based Survey; LightGBM: light gradient-boosting machine; Ungdata: Norwegian nationwide Ungdata surveys; YRBS: Youth Risk Behavior Survey.
Figure 2. Model architecture. The original Korea Youth Risk Behavior Web-based Survey (KYRBS) data set was partitioned into the original data set for model development, with performance assessed using area under the receiver operating characteristic (AUROC) curve scores. Selected high-performing models were further validated using an external Youth Risk Behavior Survey (YRBS) data set. The validation results were derived from the original data set, external results from the additional YRBS data set, and the Norwegian nationwide Ungdata survey (Ungdata) data set. CV: cross validation.

The distribution of age in the initial training cohort for KYRBS from South Korea, which was used to build the prediction model, was as follows: aged 13 years (94,923/566,875, 16.74%), aged 14 years (98,624/566,875, 17.4%), aged 15 years (100,490/566,875, 17.73%), aged 16 years (90,057/566,875, 15.89%), aged 17 years (92,071/566,875, 16.24%), and aged 18 years (90,710/566,875, 16%). For the external validation cohort using YRBS from the United States, the age distribution was as follows: aged 13 years (351/103,874, 0.34%), aged 14 years (21,095/103,874, 20.31%), aged 15 years (28,016/103,874, 26.97%), aged 16 years (25,929/103,874, 24.96%), aged 17 years (22,405/103,874, 21.57%), and aged 18 years (6078/103,874, 5.85%). Another external validation stage using Ungdata from Norway had the following age distribution: aged 13 years (5039/19,574, 25.74%), aged 14 years (4874/19,574, 24.9%), aged 15 years (5034/19,574, 25.72%), aged 16 years (3181/19,574, 16.25%), aged 17 years (845/19,574, 4.32%), and aged 18 years (601/19,574, 3.07%) (Table 1).

Table 1. Demographic characteristics of Korea Youth Risk Behavior Web-based Survey (KYRBS) data from South Korea (2009-2021), Youth Risk Behavior Survey (YRBS) data from the United States (2021), and University National General Survey (Ungdata) from Norway (2017-2019).
CharacteristicsKYRBS (n=566,875)YRBS (n=103,874)Ungdata (n=19,574)
Region, n (%)

Urban259,453 (45.77)N/AaN/A

Rural307,422 (54.23)N/AN/A
Age (years), n (%)

1394,923 (16.74)351 (0.34)5039 (25.74)

1498,624 (17.4)21,095 (20.31)4874 (24.9)

15100,490 (17.73)28,016 (26.97)5034 (25.72)

1690,057 (15.89)25,929 (24.96)3181 (16.25)

1792,071 (16.24)22,405 (21.57)845 (4.32)

1890,710 (16)6078 (5.85)601 (3.07)
Sex, n (%)

Male289,311 (51.04)51,842 (49.91)9812 (50.13)

Female277,564 (48.96)52,032 (50.09)9762 (49.87)
BMIb, n (%)

Underweight44,539 (7.86)12,541 (12.07)N/A

Normal423,384 (74.69)47,592 (45.82)N/A

Overweight48,951 (8.64)14,439 (13.9)N/A

Obese50,001 (8.82)29,302 (28.21)N/A
Academic achievement, n (%)

Low (0-19th percentiles)56,799 (10.02)N/AN/A

Lower-middle (20th -39th percentiles)132,774 (23.42)N/AN/A

Middle (40th-59th percentiles)162,447 (28.66)N/AN/A

Upper-middle (60th-79th percentiles)146,401 (25.83)N/AN/A

High (80th-100th percentiles)68,454 (12.08)N/AN/A
Household income, n (%)

Low (0-19th percentiles)19,200 (3.39)N/A8466 (43.25)

Lower-middle (20th -39th percentiles)82,314 (14.52)N/A6557 (33.5)

Middle (40th-59th percentiles)274,093 (48.35)N/A3523 (18)

Upper-middle (60th-79th percentiles)148,848 (26.26)N/A818 (4.18)

High (80th-100th percentiles)42,420 (7.48)N/A210 (1.07)
Smoking status, n (%)

Nonsmoker467,707 (82.51)84,984 (81.81)16,594 (84.78)

Smoker99,168 (17.49)18,890 (18.19)2980 (15.22)
Alcohol consumption, n (%)

Nondrinker478,305 (84.38)76,571 (73.72)9673 (49.42)

Drinker88,570 (15.62)27,303 (26.28)9901 (50.58)
Stress statusc, n (%)

Low to moderate337,938 (59.61)70,817 (68.18)N/A

High to severe228,937 (40.39)33,057 (31.82)N/A
Sadness and despair, n (%)

Low to moderate401,253 (70.78)64,190 (61.8)14,522 (74.19)

High to severe165,622 (29.22)39,684 (38.2)5052 (25.81)
Exercise status, n (%)

Not enough496,475 (87.58)87,510 (84.25)N/A

Enough70,400 (12.42)16,364 (15.75)N/A
Suicide thinking in the past year, n (%)

No482,613 (85.14)80,561 (77.56)17,652 (90.18)

Yes84,262 (14.86)23,313 (22.44)1922 (9.82)
Screentime status, n (%)

Low to moderate165,382 (29.17)27,569 (26.54)994 (5.08)

High to severe401,493 (70.83)76,305 (73.46)18,580 (94.92)

aN/A: not applicable.

bBMI was divided into 4 groups according to the National Growth Charts: underweight (0-4th percentiles), normal (5th-84th percentiles), overweight (85th-94th percentiles), and obese (95th-100th percentiles).

cStress was defined by receipt of mental health counseling owing to stress.

Both the initial training cohort and the external validation cohorts took into account socioeconomic backgrounds, such as household income and academic achievement, as well as risk behaviors such as alcohol consumption, smoking, and screen time. Additionally, factors that could potentially influence mental health, such as feelings of sadness and despair, were also considered. Inconsistencies or missing values in validation sets were addressed by implementing median imputation from the primary training data. Such thorough demographic incorporation bolsters our model performance, offering a nuanced understanding of suicidal thinking in adolescents.

ML Model Results

Table S1 in Multimedia Appendix 1 and Table 2 present the process of hyperparameter tuning the XGBoost model and the evaluation of our models conducted on data sets from 3 distinct countries using 5 performance metrics. Notably, XGBoost emerged as the frontrunner among the 4 tested models by getting hyperparameters of booster: gbtree, eval_metric: logloss, learning_rate: 0.08, max_depth: 5, n_estimators: 350, scale_pos_weight: 2, subsample: 0.09 (Table S1 in Multimedia Appendix 1). When put to the training on the KYRBS data set from South Korea, with a 95% CI, the XGBoost model reported an AUROC of 90.06 (95% CI 89.97-90.16), sensitivity was 82.11 (95% CI 81.67-82.55), specificity was 82.16 (95% CI 81.68-82.63), accuracy was 82.13 (95% CI 82.01-82.26), and balanced accuracy was 82.13 (95% CI 82.01-82.26), consistently displaying superior results compared to the other 3 models. During the external validation assessment, the model evaluation was conducted without considering the 95% CI. For the external model validation, using the YRBS from the United States, the XGBoost model achieved an AUROC of 83.09%, sensitivity of 80.26%, specificity of 75.52%, accuracy of 76.58%, and balanced accuracy of 77.89%. For the external validation using the Ungdata from Norway, the XGBoost model achieved an AUROC of 81.27%, sensitivity of 79.19%, specificity of 80%, accuracy of 79.92%, and balanced accuracy of 79.60%. Across all data sets, XGBoost consistently outperformed all other models with the highest AUROC score, which was selected as the most optimal model (Figure 3).

Table 2. Various performance metrics for 4 different machine learning algorithms in the Korea Youth Risk Behavior Web-based Survey (KYRBS) data set for validation, the Youth Risk Behavior Survey (YRBS) data set for extra validation, and another external validation at the Norwegian nationwide University National General Survey (Ungdata) surveys data set.
Data set and modelAUROCaSensitivitySpecificityAccuracyBalanced accuracy
South Korea, % (95% CI)

XGBoost90.06 (89.97-90.16)82.11 (81.67-82.55)82.16 (81.68-82.63)82.13 (82.01-82.26)82.13 (82.01-82.26)

AdaBoostb85.26 (85.14-85.38)78.08 (77.94-78.21)78.09 (77.96-78.22)78.08 (77.95-78.21)78.08 (77.95-78.21)

LightGBMc84.64 (84.53-84.75)78.96 (78.88-79.04)78.96 (78.88-79.04)78.96 (78.88-79.04)78.96 (78.88-79.04)

Random forest84.10 (83.99-84.21)76.91 (76.73-77.10)77.06 (76.98-77.15)76.99 (76.86-77.11)76.99 (76.86-77.11)
United States, %




Random forest83.2078.2877.4377.6277.86
Norway, %




Random forest79.2379.5079.5079.5579.97

aAUROC: area under the receiver operating characteristic.

bAdaBoost: adaptive boosting.

cLightGBM: light gradient-boosting machine.

Figure 3. Receiver operating characteristic (ROC) curves with CIs for the Korea Youth Risk Behavior Web-based Survey (KYRBS), Youth Risk Behavior Survey (YRBS), and University National General Survey (Ungdata) of the XGBoost model.

Feature Importance

Table 3 shows the feature importance derived from the XGBoost model, illustrating the relative contributions of each feature to predicting suicidal thinking. Notably, feelings of sadness and despair emerge as the most dominant predictor, accounting for 57.4% of the influence, followed by stress status at 19.8%. Subsequent factors include age (5.7%), household income (4%), academic achievement (3.4%), sex (2.1%), and others contributing less than 2% each.

Table 3. Feature importance of XGBoost.
FeatureImportance, %
Sadness and despair57.4
Stress status19.8
Household income4
Academic achievement3.4
Smoking status1.7
Alcohol consumption1.3
Exercise status1.3
Screentime status1

SHAP Value

We addressed a deeper visual interpretation of the SHAP values within our ML model (Figure S2 in Multimedia Appendix 1) [29]. Figure S3 in Multimedia Appendix 1 provides a waterfall plot, distinctively showcasing the cumulative contribution of each feature to a single prediction. We interpreted individual predictions by starting from the initial estimate and sequentially incorporating the influence of each feature to reach the final prediction. E[f(x)] refers to the average predicted output of the model across the entire data set, providing insights into the model’s overall prediction tendency. The starting point of the illustration, denoted as E[f(x)]=0.83, represents the model’s average prediction for the given data set. Among the variables, sadness and despair stood out, boosting the prediction by 1.17 and ranking as the most influential factor. Conversely, stress status and sex reduced the prediction by 0.86 and 0.22, respectively. This visualization offers a clear insight into the profound influence each feature wields in predicting adolescent suicidal thinking. Our ML model notably underscores a substantial reliance on the sadness, despair, and stress status features (Figure S3 in Multimedia Appendix 1).

Code Availability

Based on the results of the ML model, we established a web-based app for policy implementation or health system management to support in their decision-making process for cases involving suicidal thinking prediction use in adolescents [30]. An example of a web interface and the results are shown in Figure S4 in Multimedia Appendix 1. Custom code for the website is available on the web [31].

Key Findings

Our research represents a pioneering machine-learning initiative for predicting suicidal thinking among adolescents. We sourced distinct data sets from South Korea (KYRBS), the United States (YRBS), and Norway (Ungdata). This provided comprehensive analysis related to socioeconomic indicators and key mental health influencers such as alcohol consumption, smoking status, and feelings of sadness and despair. Importantly, our findings highlight XGBoost as the optimal predictive model, achieving an AUROC of 88.6% with the KYRBS data set. External validation with the data from the United States and Norway yielded AUROCs of 82.9% and 83.6%, respectively. The most significant predictor of suicidal thinking was sadness and despair, with a feature influence of 61%, followed by stress status at 19.6%. Using SHAP values, we further emphasized the pivotal roles of sadness, despair, and stress in predicting suicidal thinking in adolescents. To enhance the practical application of our research, we have developed a web-based platform to visualize the prediction model, accompanied by a mobile interface, enhancing its accessibility and user experience. This dual-platform system provides a more methodical and analytical approach for the public to comprehend and manage potential suicidal concerns.

Plausible Mechanism

The close relationship between feelings of sadness and despair and suicidal thinking in adolescents can be understood from various perspectives, encompassing biological and environmental factors [32]. During adolescence, the brain undergoes significant development, especially in the prefrontal cortex, which controls impulses and emotions [33]. Persistent sadness can interfere with adolescent brain development, resulting in a perpetual state of negative emotions. This increases their risk of suicidal thinking due to feelings of despair and impulsive actions [34].

The influence of external portrayals, be it from peers or the media, cannot be underestimated. When adolescents confront additional adversities, such as bullying, social isolation, or academic failures, these inherent stressors are amplified. Adolescents exposed to narratives associating despair with suicidal behaviors might inadvertently absorb these sentiments. This phenomenon, known as “suicide contagion,” postulates that exposure to others’ suicidal actions can reshape an individual’s perspective [35]. The confluence of these environmental stressors and limited emotional regulation capacities heightens their vulnerability, potentially leading them to view suicide as a viable solution to their emotional turmoil [36].

Furthermore, due to their developmental stage, many adolescents have not yet acquired the necessary emotional coping strategies [37]. When faced with intense stressors without these tools, some may come to view suicide as their only way to escape from increasingly desperate circumstances.

This emotional vulnerability is further compounded by physiological changes. The stress response, regulated by the hypothalamic-pituitary-adrenal axis, is intensified during adolescence [38]. Heightened sensitivity of the hypothalamic-pituitary-adrenal axis results in increased cortisol production in response to stress [39]. Chronic exposure to these elevated cortisol levels not only exacerbates feelings of sadness and despair but also directly contributes to an increased vulnerability to suicidal thoughts [40]. Understanding these relationships makes it evident that both the emotional responses induced by stress and the physiological effects of stress significantly influence the increased propensity for suicidal thinking during this critical phase of life.

Strengths and Limitations

The limitations of this study should be stated. One primary concern pertains to the use of self-reported data, which exposes our results to potential biases, such as recall and social desirability. While this approach offers insights directly from the participants, such susceptibilities might skew the data and ultimately affect the model’s performance. It is also worth noting that the foundational training data were sourced predominantly from adolescents in South Korea [41,42]. This could amplify specific cultural or racial attributes distinctive to Korean adolescents. Equally important to note is that establishing a direct cause-and-effect relationship between significant risk factors and adolescent suicidal thinking remains elusive. This study, while expansive, does not determine if suicidal thinking is a cause or an effect of other risk factors. Further research is needed to unravel these complex interconnections. The potential for overfitting is another critical limitation to consider. Our comprehensive model, regardless of its use of 10-fold cross-validation, might inadvertently capture anomalies rather than genuine patterns [43]. Furthermore, our method for managing missing data, especially through median imputation, poses the risk of introducing unintended biases, which could impact the model’s performance [44]. Lastly, while predicting suicide attempts rather than suicide ideation may be crucial in suicide-related research, the low prevalence of suicide attempts presents limitations in constructing ML models: thus, we have developed predictive models for suicide ideation. With our predictive model, policy researchers, physicians, and community neighbors can develop individualized prevention strategies for these adolescents.

Despite these limitations, the strengths of this study are manifold. Notably, the SHAP value analysis highlights the importance of diverse demographic and behavioral indicators in understanding suicidal thinking among adolescents. This study provides nuanced insight into each feature’s unique influence and the model’s decision-making process [29]. The robustness of the model is also noteworthy, as evidenced by its uniform effectiveness across data sets spanning South Korea, the United States, and Norway. Such wide-reaching efficacy suggests the adaptability of the model to different cultural and demographic landscapes. Another strength lies in the real-world applicability of our research. The development of our advanced web-based platform and mobile interface marks a notable advancement in practical application. By offering a user-friendly interface tailored for both desktop and mobile users, we enhance accessibility and promote greater self-awareness about suicidal risk. This can encourage individuals to seek timely professional assistance or supportive resources, serving as a preventive measure against severe mental health crises [45].

Clinical and Policy Implications

In light of the findings, several crucial policy implications emerge. Foremost, the significant role of sadness and despair as predictors underscores the necessity to prioritize mental health support for adolescents [46]. This prominence not only necessitates immediate interventions but also stresses the vital role of education and awareness initiatives, targeting both risk behaviors and associated mental health ramifications. Early identification of suicidal thinking is paramount [47]. Recognizing these indications enables health care professionals to initiate early intervention strategies. This preemptive approach should include tailored counseling, support group engagements, or intensive therapeutic interventions. This can prevent the progression toward actual suicide attempts, which might be driven by mixed emotions or even an intent just to signal distress [48]. Moreover, there is a pressing need to bolster educational and awareness campaigns concerning suicide. Such campaigns serve to equip adolescents with the tools and knowledge necessary, encouraging them to navigate challenges related to risky behaviors and maintain positive mental health perspectives [49].


In addressing the pressing global concern of adolescent suicide, this study used ML to offer novel insights into preemptive detection. By integrating diverse data sets across 3 nations, the study highlighted the superiority of the XGBoost model in predicting suicidal thinking, achieving remarkable AUROCs of 90.06% (95% CI 89.97-90.16; KYRBS from South Korea; discovery), 83.09% (YRBS from the United States; extra validation), and 81.27% (Ungdata from Norway; extra validation). Our findings emphasize the significant role of emotional health indicators in predicting suicidal thinking among adolescents. Specifically, sadness and despair proved to be the most influential predictors, followed by stress status and age. Through our robust, cross-culturally validated model and its accessibility through web-based platforms, we underscore the potential for timely interventions and offer a promising blueprint for future mental health strategies and preventive measures for at-risk adolescents.


This research was supported by a grant from the Korea Health Technology R&D Project through the Korea Health Industry Development Institute, funded by the Ministry of Health & Welfare, Republic of Korea (HI22C1976). The funders had no role in study design, data collection, data analysis, data interpretation, or the writing of the report.

Data Availability

The data sets generated during and/or analyzed during this study are available from the corresponding author upon reasonable request. The study protocol and statistical code are available from DKY. The data set can be accessed from the Korean Disease Control and Prevention Agency, US Centers for Disease Control and Prevention, and Norwegian Social Research Institute (NOVA) through a data use agreement.

Authors' Contributions

DKY had full access to all of the data in the study and took responsibility for the integrity of the data and the accuracy of the data analysis. HK, YS, Hojae Lee, RK, SW, and DKY were responsible for study concept and design; acquisition, analysis, or interpretation of data; drafting of the manuscript; and statistical analysis. All authors were responsible for critical revision of the manuscript for important intellectual content and approved the final version before submission. Study supervision was done by RK, SW, and DKY. DKY supervised the study and is the guarantor for this study. HK, YS, and Hojae Lee contributed equally as first authors. RK, SW, and DKY contributed equally as corresponding authors. The corresponding author attests that all listed authors meet authorship criteria and that no others meeting the criteria have been omitted.

Conflicts of Interest

None declared.

Multimedia Appendix 1

Study population; SHAP value of XGBoost model; SHAP waterfall plot; deployed web-based application to provide suicidal thinking prediction among adolescents; process of hyperparameter tuning for the XGBoost model on KYRBS data set; and transparent reporting of a multivariable prediction model for individual.

DOCX File , 410 KB

  1. Hughes JL, Horowitz LM, Ackerman JP, Adrian MC, Campo JV, Bridge JA. Suicide in young people: screening, risk assessment, and intervention. BMJ. 2023;381:e070630. [FREE Full text] [CrossRef] [Medline]
  2. Lee H, Cho JK, Park J, Lee H, Fond G, Boyer L, et al. Machine learning-based prediction of suicidality in adolescents with allergic rhinitis: derivation and validation in 2 independent nationwide cohorts. J Med Internet Res. 2024;26:e51473. [FREE Full text] [CrossRef] [Medline]
  3. Hawton K, Hill NTM, Gould M, John A, Lascelles K, Robinson J. Clustering of suicides in children and adolescents. Lancet Child Adolesc Health. 2020;4(1):58-67. [CrossRef] [Medline]
  4. India State-Level Disease Burden Initiative Suicide Collaborators. Gender differentials and state variations in suicide deaths in India: the Global Burden of Disease Study 1990-2016. Lancet Public Health. 2018;3(10):e478-e489. [FREE Full text] [CrossRef] [Medline]
  5. Méndez-Bustos P, Fuster-Villaseca J, Lopez-Castroman J, Jiménez-Solomon O, Olivari C, Baca-Garcia E. Longitudinal trajectories of suicidal ideation and attempts in adolescents with psychiatric disorders in Chile: study protocol. BMJ Open. 2022;12(2):e051749. [FREE Full text] [CrossRef] [Medline]
  6. Mann JJ, Michel CA, Auerbach RP. Improving suicide prevention through evidence-based strategies: a systematic review. Am J Psychiatry. 2021;178(7):611-624. [FREE Full text] [CrossRef] [Medline]
  7. Mars B, Heron J, Klonsky ED, Moran P, O'Connor RC, Tilling K, et al. Predictors of future suicide attempt among adolescents with suicidal thoughts or non-suicidal self-harm: a population-based birth cohort study. Lancet Psychiatry. 2019;6(4):327-337. [FREE Full text] [CrossRef] [Medline]
  8. Kwak CW, Ickovics JR. Adolescent suicide in South Korea: risk factors and proposed multi-dimensional solution. Asian J Psychiatr. 2019;43:150-153. [CrossRef] [Medline]
  9. Jeong SC, Kim JY, Choi MH, Lee JS, Lee JH, Kim CW, et al. Identification of influencing factors for suicidal ideation and suicide attempts among adolescents: 11-year national data analysis for 788,411 participants. Psychiatry Res. 2020;291:113228. [CrossRef] [Medline]
  10. Mayes TL, Carmody T, Rush AJ, Nandy K, Emslie GJ, Kennard BD, et al. Predicting suicidal events: a comparison of the Concise Health Risk Tracking Self-Report (CHRT-SR) and the Columbia Suicide Severity Rating Scale (C-SSRS). Psychiatry Res. 2023;326:115306. [FREE Full text] [CrossRef] [Medline]
  11. Kim N, Song JY, Yang H, Kim MJ, Lee K, Shin YH, et al. National trends in suicide-related behaviors among youths between 2005-2020, including COVID-19: a Korean representative survey of one million adolescents. Eur Rev Med Pharmacol Sci. 2023;27(3):1192-1202. [FREE Full text] [CrossRef] [Medline]
  12. Park H, Lee K. Using boosted machine learning to predict suicidal ideation by socioeconomic status among adolescents. J Pers Med. 2022;12(9):1357. [FREE Full text] [CrossRef] [Medline]
  13. Czyz EK, Koo HJ, Al-Dajani N, King CA, Nahum-Shani I. Predicting short-term suicidal thoughts in adolescents using machine learning: developing decision tools to identify daily level risk after hospitalization. Psychol Med. 2023;53(7):2982-2991. [FREE Full text] [CrossRef] [Medline]
  14. Weller O, Sagers L, Hanson C, Barnes M, Snell Q, Tass ES. Predicting suicidal thoughts and behavior among adolescents using the risk and protective factor framework: a large-scale machine learning approach. PLoS One. 2021;16(11):e0258535. [FREE Full text] [CrossRef] [Medline]
  15. Oh J, Lee M, Lee H, Yang H, Park J, Rahmati M, et al. Hand and oral hygiene practices of South Korean adolescents before and during the COVID-19 pandemic. JAMA Netw Open. 2023;6(12):e2349249. [FREE Full text] [CrossRef] [Medline]
  16. Oh J, Kim M, Rhee SY, Rahmati M, Koyanagi A, Smith L, et al. National trends in the prevalence of screen time and its association with biopsychosocial risk factors among korean adolescents, 2008-2021. J Adolesc Health. 2024;74(3):504-513. [CrossRef] [Medline]
  17. Wilimitis D, Turer RW, Ripperger M, McCoy AB, Sperry SH, Fielstein EM, et al. Integration of face-to-face screening with real-time machine learning to predict risk of suicide among adults. JAMA Netw Open. 2022;5(5):e2212095. [FREE Full text] [CrossRef] [Medline]
  18. Lundberg SM, Erion G, Chen H, DeGrave A, Prutkin JM, Nair B, et al. From local explanations to global understanding with explainable AI for trees. Nat Mach Intell. 2020;2(1):56-67. [FREE Full text] [CrossRef] [Medline]
  19. Lever J, Krzywinski M, Altman N. Model selection and overfitting. Nat Methods. 2016;13(9):703-704. [FREE Full text] [CrossRef]
  20. Kwon R, Lee H, Kim MS, Lee J, Yon DK. Machine learning-based prediction of suicidality in adolescents during the COVID-19 pandemic (2020-2021): derivation and validation in two independent nationwide cohorts. Asian J Psychiatr. 2023;88:103704. [CrossRef] [Medline]
  21. Kim J, Kim SC, Kang D, Yon DK, Kim JG. Classification of Alzheimer's disease stage using machine learning for left and right oxygenation difference signals in the prefrontal cortex: a patient-level, single-group, diagnostic interventional trial. Eur Rev Med Pharmacol Sci. 2022;26(21):7734-7741. [FREE Full text] [CrossRef] [Medline]
  22. Lee SW. Methods for testing statistical differences between groups in medical research: statistical standard and guideline of Life Cycle Committee. Life Cycle. 2022;2:e1. [FREE Full text] [CrossRef]
  23. Kim MS, Lee H, Lee SW, Kwon R, Rhee SY, Lee JA, et al. Long-term autoimmune inflammatory rheumatic outcomes of COVID-19 : a binational cohort study. Ann Intern Med. 2024;177(3):291-302. [FREE Full text] [CrossRef] [Medline]
  24. Yoo HW, Jin HY, Yon DK, Effenberger M, Shin YH, Kim SY, et al. Non-alcoholic fatty liver disease and COVID-19 susceptibility and outcomes: a Korean nationwide cohort. J Korean Med Sci. 2021;36(41):e291. [FREE Full text] [CrossRef] [Medline]
  25. Lee SW, Kim SY, Moon SY, Yang JM, Ha EK, Jee HM, et al. Estimating COVID-19 infection and severity risks in patients with chronic rhinosinusitis: a Korean nationwide cohort Study. J Allergy Clin Immunol Pract. 2021;9(6):2262-2271.e2. [FREE Full text] [CrossRef] [Medline]
  26. Barish M, Bolourani S, Lau LF, Shah S, Zanos TP. External validation demonstrates limited clinical utility of the interpretable mortality prediction model for patients with COVID-19. Nat Mach Intell. 2020;3(1):25-27. [FREE Full text] [CrossRef]
  27. Chung H, Ko H, Lee H, Yon DK, Lee WH, Kim T, et al. Development and validation of a deep learning model to diagnose COVID-19 using time-series heart rate values before the onset of symptoms. J Med Virol. 2023;95(2):e28462. [CrossRef] [Medline]
  28. Kim J, Lee H, Lee J, Rhee SY, Shin JI, Lee SW, et al. Quantification of identifying cognitive impairment using olfactory-stimulated functional near-infrared spectroscopy with machine learning: a post hoc analysis of a diagnostic trial and validation of an external additional trial. Alzheimers Res Ther. 2023;15(1):127. [FREE Full text] [CrossRef] [Medline]
  29. Chen H, Lundberg SM, Lee SI. Explaining a series of models by propagating Shapley values. Nat Commun. 2022;13(1):4512. [FREE Full text] [CrossRef] [Medline]
  30. Probability for suicidal thinking. Streamlit. URL: [accessed 2024-04-28]
  31. CenterForDH/suicidality. GitHub. URL: https:// [accessed 2024-04-28]
  32. Malhi GS, Mann JJ. Depression. Lancet. 2018;392(10161):2299-2312. [CrossRef] [Medline]
  33. Kolk SM, Rakic P. Development of prefrontal cortex. Neuropsychopharmacology. 2022;47(1):41-57. [FREE Full text] [CrossRef] [Medline]
  34. Straub J, Brown R, Malejko K, Bonenberger M, Grön G, Plener PL, et al. Adolescent depression and brain development: evidence from voxel-based morphometry. J Psychiatry Neurosci. 2019;44(4):237-245. [FREE Full text] [CrossRef] [Medline]
  35. Khasawneh A, Madathil KC, Dixon E, Wiśniewski P, Zinzow H, Roth R. Examining the self-harm and suicide contagion effects of the blue whale challenge on YouTube and Twitter: qualitative study. JMIR Ment Health. 2020;7(6):e15973. [FREE Full text] [CrossRef] [Medline]
  36. Shapero BG, Farabaugh A, Terechina O, DeCross S, Cheung JC, Fava M, et al. Understanding the effects of emotional reactivity on depression and suicidal thoughts and behaviors: moderating effects of childhood adversity and resilience. J Affect Disord. 2019;245:419-427. [CrossRef] [Medline]
  37. Gruhn MA, Compas BE. Effects of maltreatment on coping and emotion regulation in childhood and adolescence: a meta-analytic review. Child Abuse Negl. 2020;103:104446. [CrossRef] [Medline]
  38. Häusl AS, Brix LM, Hartmann J, Pöhlmann ML, Lopez J, Menegaz D, et al. The co-chaperone Fkbp5 shapes the acute stress response in the paraventricular nucleus of the hypothalamus of male mice. Mol Psychiatry. 2021;26(7):3060-3076. [FREE Full text] [CrossRef] [Medline]
  39. Russell G, Lightman S. The human stress response. Nat Rev Endocrinol. 2019;15(9):525-534. [CrossRef] [Medline]
  40. Thoma MV, Rohleder N, Rohner SL. Clinical ecopsychology: the mental health impacts and underlying pathways of the climate and environmental crisis. Front Psychiatry. 2021;12:675936. [FREE Full text] [CrossRef] [Medline]
  41. Kim Y, Choi S, Chun C, Park S, Khang YH, Oh K. Data resource profile: the Korea Youth Risk Behavior Web-based Survey (KYRBS). Int J Epidemiol. 2016;45(4):1076-1076e. [FREE Full text] [CrossRef] [Medline]
  42. Woo HG, Park S, Yon H, Lee SW, Koyanagi A, Jacob L, et al. National trends in sadness, suicidality, and COVID-19 pandemic-related risk factors among South Korean adolescents from 2005 to 2021. JAMA Netw Open. 2023;6(5):e2314838. [FREE Full text] [CrossRef] [Medline]
  43. Valavi R, Elith J, Lahoz‐Monfort JJ, Guillera‐Arroita G. BlockCV: an R package for generating spatially or environmentally separated folds for k-fold cross-validation of species distribution models. Methods Ecol Evol. 2018;10(2):225-232. [FREE Full text] [CrossRef]
  44. Yang J, Soltan AAS, Eyre DW, Clifton DA. Algorithmic fairness and bias mitigation for clinical machine learning with deep reinforcement learning. Nat Mach Intell. 2023;5(8):884-894. [FREE Full text] [CrossRef] [Medline]
  45. Dinh-Le C, Chuang R, Chokshi S, Mann D. Wearable health technology and electronic health record integration: scoping review and future directions. JMIR Mhealth Uhealth. 2019;7(9):e12861. [FREE Full text] [CrossRef] [Medline]
  46. Rogers A, Obst S, Teague SJ, Rossen L, Spry EA, Macdonald JA, et al. Association between maternal perinatal depression and anxiety and child and adolescent development: a meta-analysis. JAMA Pediatr. 2020;174(11):1082-1092. [FREE Full text] [CrossRef] [Medline]
  47. Stanley B, Mann JJ. The need for innovation in health care systems to improve suicide prevention. JAMA Psychiatry. 2020;77(1):96-98. [CrossRef] [Medline]
  48. Robinson J, Bailey E, Witt K, Stefanac N, Milner A, Currier D, et al. What works in youth suicide prevention? A systematic review and meta-analysis. EClinicalMedicine. 2018;4-5:52-91. [FREE Full text] [CrossRef] [Medline]
  49. Thorn P, Hill NT, Lamblin M, Teh Z, Battersby-Coulter R, Rice S, et al. Developing a suicide prevention social media campaign with young people (The #Chatsafe Project): co-design approach. JMIR Ment Health. 2020;7(5):e17520. [FREE Full text] [CrossRef] [Medline]

AdaBoost: adaptive boosting
AUROC: area under the receiver operating characteristic
KYRBS: Korea Youth Risk Behavior Web-based Survey
LightGBM: light gradient-boosting machine
ML: machine learning
SHAP: Shapley additive explanations
TRIPOD: Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis
Ungdata: University National General Survey
YRBS: Youth Risk Behavior Survey

Edited by Y Hong; submitted 29.12.23; peer-reviewed by F Alam, D Brent; comments to author 19.03.24; revised version received 24.03.24; accepted 25.03.24; published 17.05.24.


©Hyejun Kim, Yejun Son, Hojae Lee, Jiseung Kang, Ahmed Hammoodi, Yujin Choi, Hyeon Jin Kim, Hayeon Lee, Guillaume Fond, Laurent Boyer, Rosie Kwon, Selin Woo, Dong Keon Yon. Originally published in the Journal of Medical Internet Research (, 17.05.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.