Our study was prospectively registered on PROSPERO (ID: CRD42024566433). The study followed the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [24] (Multimedia Appendix 1).

The inclusion criteria were as follows: (1) studies involving minors as the research subjects and (2) studies that developed a complete ML model for predicting suicide-related behaviors. Moreover, studies that performed only internal validation without external validation were also considered for inclusion in this systematic review.

The exclusion criteria were as follows: (1) conference abstracts, brief reports, or other publications that were not peer-reviewed, (2) studies only analyzing risk or predictive factors for suicide-related behaviors without a complete ML model, (3) studies with a sample size of fewer than 20 cases, and (4) studies that did not report any measures of model accuracy (receiver operating characteristic curve, sensitivity, specificity, confusion matrix, accuracy, F₁-score, and calibration curve).

PubMed, Web of Science, Cochrane, and Embase databases were systematically searched up to April 20, 2024. The strategy involved both subject headings and free-text terms, covering terms such as “adolescent,” “suicide,” and “ML.” To ensure global representativeness and the quality of the research, no regional or time-related restrictions were imposed. The detailed search strategy is illustrated in Multimedia Appendix 2.

The selection and screening of studies were independently performed by 2 researchers (LL and ZL). In cases of discrepancies during the screening process, consensus was reached by team discussion or consultation with a third expert (GH). The retrieved articles were uploaded to EndNote. After duplicates were removed, the titles and abstracts of the remaining articles were checked. Full-text articles were downloaded and further screened to obtain eligible articles. Additionally, references to the selected articles were examined to identify any potentially missed articles.

A standardized data extraction spreadsheet was used to collect key information, including the first author, publication year, author country, study type, patient source, population, predicted event, event definition, number of predicted events, total cases, number of predicted events in the training or validation set, total cases in the training or validation set, method of validation set generation, overfitting method, prediction settings for event number, validation set case number, missing data handling method, variable selection method, model type, and modeling variables. Data extraction was performed independently by 2 researchers (LL and ZL) who cross-checked their work. Disputes, if any, were settled by team discussion or consultation with a third expert (GH).

PROBAST (Prediction Model Risk of Bias Assessment Tool), a standard quality evaluation tool [25], was employed to assess the risk of bias in models from the original studies. This tool comprises a series of questions across 4 distinct domains: participant, predictor, outcome, and statistical analysis, presenting the overall risk of bias and applicability. These domains include 2, 3, 6, and 9 questions, respectively, each of which can be answered in 1 of 3 ways: yes/possibly yes, no/possibly no, or no information. A domain was considered at high risk if at least one question was answered with “no” or “possibly no.” A domain was considered at low risk if all questions were answered with “yes” or “possibly yes.” The overall risk of bias was deemed low if all domains were rated as low risk, while the overall risk was considered high if at least one domain was rated as high risk. Two researchers (LL and YH) independently appraised the risk of bias based on PROBAST by carefully reviewing the methodological section of each paper. Following this, cross-checking was performed, and in cases of disagreement, a third researcher (GH) was consulted for determination.

In this study, suicide events were categorized into 5 subgroups: NSSI, suicide attempts, suicidal ideation, suicide attempts combined with suicidal ideation, and suicide attempts combined with NSSI. Subgroup analyses of the ML models based on the type of predicted event were performed to examine the impact of specific events on effect sizes.

A meta-analysis was performed on the area under the receiver operating characteristic curve (AUC), a metric for evaluating the overall accuracy of ML models. In certain primary studies where the c-index lacked 95% CI and standard error, the standard error was estimated based on the work of Debray et al [26]. Heterogeneity across studies was assessed via the I² statistic. When the I² value was >50%, a random effects model was adopted to summarize the AUC, and when the I² value was <50%, a fixed effects model was used. A meta-analysis was conducted solely on the validation set of the models. We have discussed the performance of ML in forecasting NSSI, suicide attempts, suicidal ideation, suicide attempts combined with suicidal ideation, and suicide attempts combined with NSSI.

Moreover, a meta-analysis was conducted on the sensitivity and specificity of ML in predicting suicide-related events using diagnostic 2×2 tables. A bivariate mixed effects model was employed. In cases where many primary studies did not provide 2×2 tables, the sensitivity, specificity, precision, and accuracy data provided in the original studies, along with case numbers, were employed to estimate the diagnostic 2×2 table. A P value of <.05 was considered to indicate statistical significance.

This study initially identified 4732 published articles, and 648 duplicate references were discarded automatically and manually via EndNote. The titles and abstracts of the remaining 4084 articles were reviewed, and 4030 articles were deleted, including studies unrelated to the research topic and ineligible studies, such as reviews, guidelines, conference abstracts, case reports, replies, and letters. For potentially eligible studies, the full texts were obtained and carefully evaluated. After a full-text review, 12 articles were removed. Of these 12 articles, 4 were unpublished in full, 3 involved controversial definitions of suicide outcomes, and 5 conducted only differential factor analyses without constructing models. Ultimately, 42 articles [27-68] were included in this study (Figure 1).

The 42 articles included 1,408,375 adolescents aged 11 to 20 years. While there were slight variations in the age groups across different studies, the overall coverage encompassed the critical stages from adolescence to early adulthood. The analysis incorporated 104 distinct types of ML models from articles published between 2018 and 2024. Among the studies, 23 involved single-center data, 11 involved multi-center data, and 8 involved registry databases. The studies covered various countries and regions, including North America, Europe, and Asia, with a higher number of articles from the United States (n=11), South Korea (n=9), and China (n=8). The types of studies in this review were diverse, and there were 9 cross-sectional studies, 22 prospective cohort studies, 7 retrospective cohort studies, and 4 case-control studies. Among the studies, 16 considered suicide attempts combined with suicidal ideation as the outcome event, 9 focused on suicide attempts, and 7 focused on suicidal ideation. Moreover, 7 studies regarded NSSI as a predictive event for suicide, and 3 combined NSSI with suicide attempts. Most studies employed internal validation, with k-fold cross-validation used in 35 studies and bootstrapping used in 2 studies. Only 5 studies performed external validation. The most common validation methods used were k-fold cross-validation and bootstrap sampling to prevent model overfitting. Of the 42 studies, 37 performed multivariable analysis to screen for high-risk factors and 5 employed both univariate and multivariate analyses. The ML algorithms included support vector machine (SVM), random forest (RF), artificial neural network (ANN), logistic regression (LR), gradient boosting machine (GBM), decision tree (DT), and extreme gradient boosting (XGBoost) (Table 1).

The risk of bias in 104 suicide prediction models was assessed. Of these 104 models, 92 (88.5%) were deemed to have a high risk of bias and 12 (11.5%) were deemed to have a low risk of bias. Models with a high risk of bias exhibited significant discrepancies in 4 different domains. Among the models, 11 (10.6%) had a high risk of bias in the research subject domain, 11 (10.6%) had a high risk of bias in the predictor domain, 0 (0%) had a high risk of bias in the outcome domain, and 89 (85.6%) had a high risk of bias in the statistical analysis domain (Figure 2). Within the statistical analysis domain, the most common causes of an elevated risk of bias were unreasonable sample sizes (62/104, 59.6%) and improper handling of continuous and categorical independent variables (78/104, 75.0%). These findings indicate that some studies may have flaws in terms of design and lack transparency in methodology and data reliability, which could undercut the accuracy and reliability of their conclusions.

This systematic review and meta-analysis was carried out to evaluate the performance of ML in predicting the risk of suicide in adolescents. The goal was to provide a more precise risk assessment tool to assist clinicians and public health professionals in preventing suicide among adolescents. Through model interpretation and validation, ML techniques demonstrated considerable potential in predicting the risk of suicide in adolescents.

Despite the increasing focus on adolescent suicide crisis interventions in recent years, there remains a lack of consensus on how to best conceptualize these behaviors. One of the most debated issues in the existing literature is whether NSSI should be classified as a suicide-related behavior. Some studies argue that suicide attempts and NSSI are distinct, while others contend that they are complementary and strongly interconnected, both falling under the category of suicide-related behaviors [69-74]. Klonsky et al [73] acknowledged a strong correlation between NSSI and suicide attempts. A survey of adolescents in Scotland found that at least one-ninth of interviewees reported a suicide attempt, 1 in 6 reported NSSI, and 6.5% reported both behaviors [74], indicating a significant interaction between them. Ultimately, this review identified 6 studies on NSSI, and 3 studies predicted both NSSI and suicide attempts. From the perspective of preventing suicide risk, this review categorizes NSSI into suicide-related behaviors, aiming to encompass all behaviors that may potentially lead to suicidal outcomes.

ML techniques have increasingly been applied to predict suicide risk. Researchers have continuously improved the accuracy of ML in predicting the risk of suicide through multiple iterations. In recent years, several studies have examined the performance of ML models in predicting suicide-related outcomes. Studies by Wang et al [75] and Kusuma et al [76] indicate that ML can surpass traditional clinical and statistical methods, providing higher prediction accuracy. In this review, the AUC values of all models predicting various suicide outcome events exceeded 50%, and the vast majority of AUC values stabilized between 70% and 80%, demonstrating the strong discriminative power of ML models in predicting suicide-related events. The results of random effects models revealed that ML algorithms also exhibited high sensitivity and specificity in predicting suicide-related behaviors, showing clear superiority. Ehtemam et al [21] and Corke et al [77], in their reported reviews, compared ML with traditional models, focusing on the predictive value of ML models for the risk of suicide. Kusuma et al [76] used more appropriate metrics to assess the performance of ML models in the prediction of suicide-related outcomes, including ideation, attempts, and mortality. However, these studies generally relied on selected algorithms, such as RF and LR, without sufficiently justifying the rationale for these choices or systematically comparing a broader range of algorithms. As a result, there may be an overreliance on specific types of ML techniques, potentially overlooking alternative algorithms that may be more suited to particular datasets or research questions. In this context, this study, by comparing the predictive performance of various ML models, seeks to present valuable insights for selecting predictive models for different suicide outcomes.

Another notable limitation lies in the integration of diverse datasets. In most previous literature, the robustness of ML predictions often depended on the availability of large-scale, high-quality datasets. Many of the samples in these studies failed to adequately represent the broader population characteristics, thereby limiting the generalizability of their findings. Currently, no systematic review or meta-analysis has specifically focused on the prediction of suicide risk in adolescent populations as a high-risk group. This gap may stem from the limited quality and generalizability of adolescent sample data. Among the included studies on prediction models for adolescent suicide, over 50% (23 studies) used single-center data. Samaga et al [78] argued that the homogeneity of such data limits the model’s ability to learn broader features, thereby impacting its adaptability to different contexts. Furthermore, adolescent samples are often affected by issues, such as sample homogeneity, selection bias, self-reporting bias, background differences in samples, and sample attrition in longitudinal studies [79-83]. Hence, a far greater proportion of studies focus on the general population rather than adolescents as a specific research subject. Moreover, since suicide-related issues are sensitive, especially during the collection and sharing of adolescent data, which is often restricted by legal and ethical limitations, data availability is constrained. Researchers must not only face the challenges of societal sensitivity but also protect the private and personal information of participants. Obtaining sufficient suicide-related data is becoming increasingly difficult. Even when data are available, the quality of the data may be compromised due to cognitive limitations of adolescents, data generalizability issues, and potential reporting and collection errors. Through systematic reviews and meta-analyses integrating the results of all studies on the prediction of adolescent suicide, the accuracy and predictive performance of these models can be further enhanced.

In previous clinical studies based on ML, the types of models were generally simple, and traditional models, such as LR and DT, were predominant. However, among the studies included in our research, most used multiple ML approaches. This suggests that research on adolescent suicide tends to favor more complex and diverse ML techniques, particularly ensemble methods and nonlinear models. Furthermore, predicting suicide-related outcomes in adolescents using ML algorithms remains an emerging field of research. All the studies included in our analysis were conducted within the past 6 years, despite no restriction on the publication time during the literature search process. In our study, RF was the most commonly used ML model. It demonstrated notable superiority in handling imbalanced datasets, with an AUC value exceeding 80% in all 4 subgroups. A wealth of literature has also confirmed that the RF model, due to its ability to model nonlinear relationships and its insensitivity to overfitting, typically exhibits high predictive performance compared to other ML methods [84-86]. The XGBoost model also performed excellently, particularly in predicting suicide attempts, where it outperformed all other ML models with an AUC value of 0.87. In predicting other suicide-related events, the AUC value of XGBoost remained consistently around 0.8, which aligns with the findings of Ehtemam et al [21]. Additionally, Ehtemam et al [21] observed that neural network algorithms were less accurate. Unfortunately, our study did not compute the combined accuracy of this model. Hence, it is impossible to verify their findings. Nevertheless, the performance of neural networks in predicting certain suicide events in our study was impressive. In predicting suicide attempts and suicide attempts combined with suicidal ideation, ANN achieved an AUC of more than 80%. We believe that neural networks, which can handle complex nonlinear relationships, may have particular advantages in processing data related to mental health. Moreover, the AUC value of the LightGBM model was even more than 90% in the prediction of suicide attempts combined with suicidal ideation.

Regarding sensitivity and specificity, various ML models demonstrated relatively consistent results. Most models with higher sensitivity also exhibited higher specificity, with specificity higher than sensitivity. In this study, the ML model demonstrated the highest level of overall sensitivity in predicting suicide attempts, indicating its superior ability to minimize false negatives, that is, effectively identifying individuals who truly experience suicide attempts. Conversely, it exhibited the highest level of overall specificity in identifying NSSI, suggesting its excellent performance in reducing false positives. In other words, it can accurately exclude individuals who do not engage in NSSI behavior. This meta-analysis found that in suicide event subgroups containing at least four models, the XGBoost model demonstrated superior sensitivity and specificity in predicting the risk of suicide among adolescent patients. In particular, it had the highest sensitivity and specificity in predicting suicide attempts, with a sensitivity of 0.81 (95% CI 0.71-0.88) and a specificity of 0.84 (95% CI 0.77-0.89). This result indicates that the model accurately identified 81% of individuals with suicide attempts and correctly identified 84% of those without suicide attempts. This superior performance can primarily be attributed to the model’s optimization strategies. First, XGBoost uses a gradient boosting algorithm to iteratively optimize the loss function, enabling more precise capture of nonlinear relationships within the data [21]. Moreover, XGBoost supports the customization of the loss function, which allows the model to adjust its optimization objectives according to the specific task, thereby enhancing both sensitivity and specificity [21]. It was also observed that ML models, such as LightGBM, GBM, and ANN, exhibited high sensitivity and specificity. However, their application in the prediction of suicide events remains relatively limited, suggesting that these models hold considerable potential in the prediction of adolescent suicide and are expected to be broadly used in the prediction of suicide.

The significant differences in the performances of different ML models in predicting suicide-related events may be attributed to the characteristics of the models themselves. Suicide-related data often contain complex psychological, social, and behavioral characteristics, and some models may be more skillful at capturing the complex relationships between these characteristics. In addition, the size and quality of the datasets included in different studies may also affect the performance of the models. Future research can use more interpretability techniques and combine research results from psychology and sociology to better understand the reasons for differences in model performance.

In ML research, selecting an appropriate validation set generation method is crucial. In this study, there were 2 main methods for generating validation sets: internal validation and external validation. Internal validation primarily assesses the internal validity of the predictive model through techniques such as random sampling, cross-validation, and bootstrapping. In contrast, external validation evaluates the model’s ability to generalize to populations and environments that have not been explored by the dataset. By applying the model to diverse external datasets, researchers can mitigate the potential bias introduced by the original training cohort. This process helps to validate the model across various clinical pathways and population characteristics, ensuring that it is not merely reflective of the specific conditions of the training data [21]. However, internal validation remains the dominant method for validating ML models, as data barriers often complicate the interpretation of a model’s generalizability. Combining internal and external validation methods provides a more comprehensive understanding of the model’s generalization ability. This issue is especially critical in suicide prediction, which involves rare events and requires highly accurate models with strong generalizability [87,88]. Unfortunately, most of the studies included in this review predominantly employed internal validation, with a few using external validation. This may result in misleadingly optimistic accuracy estimates, raising concerns about the model’s applicability in real-world settings.

This study provides preliminary evidence for adolescent suicide and incorporates a sufficiently large sample size to ensure the robustness of the evidence. However, several limitations exist in our research. First, there was a notable degree of heterogeneity, which represents a common challenge faced by meta-analyses based on ML techniques. Subgroup analyses were carried out to explore the sources of this heterogeneity. Although most ML models demonstrated exceptional performance in identifying potential risks of suicide, the interpretability of their predictive results remains challenging. In the prediction of NSSI, suicidal ideation, suicide attempts, and combined suicidal ideation, there were considerable differences in model performance. The difference in the AUC of different models for predicting the same event was as large as 20%-30%. In contrast, all ML models exhibited high predictive efficacy for suicide attempts, with AUC values exceeding 80%. Glenn et al [89] suggested that ML models tend to perform better in predicting suicide attempts, which has a relatively low incidence rate. For the prediction of suicide attempts combined with NSSI, 3 models showed consistently good and balanced predictive performance.

Second, while the results from the validation sets were summarized, the validation methods of the models were predominantly internal validation approaches (such as k-fold cross-validation and the bootstrap method), which limits the generalizability of these models. Furthermore, since the number of eligible studies was limited, the psychological mechanisms underlying each suicide outcome or the predictive performances of ML for the risk of suicide in the context of socioeconomic factors across different suicide events were not thoroughly elucidated. Third, a 2×2 diagnostic contingency table was constructed based on sensitivity, specificity, positive predictive value, and accuracy, in conjunction with the total number of cases. The process of estimation, however, may introduce assumptions and estimation errors, potentially leading to bias in the meta-analysis. Finally, the ethical issues surrounding the use of ML for suicide prediction are often underestimated. ML models are typically regarded as “black boxes,” and concerns regarding the interpretability and transparency of their decision-making processes are particularly pronounced when predicting suicide risk. This is especially relevant when considering the potential clinical implications of such predictions [89]. Underestimating these concerns may pose risks, and if not managed appropriately, ML-driven clinical practices could inadvertently cause psychological harm to individuals or lead to societal consequences.

Through a systematic review and meta-analysis, this study explored the reliability and accuracy of ML techniques in predicting the risk of suicide in adolescents. Our findings serve as a theoretical foundation and practical pathway for the future development of more precise and tailored prevention strategies. Our study also has limitations, including the lack of external validation and the potential for overfitting. Future research should incorporate larger and more diverse datasets, along with external validation, to enhance the practicality and reliability of prediction models. Future studies are expected to further advance the development of predictive models for adolescent suicide. Researchers can develop simple tools for predicting adolescent suicide risk based on ML methods, fostering interdisciplinary collaboration, and improving data collection methods, thereby enhancing the interpretability of these models. However, prior to the application of ML models in real-world settings, rigorous ethical review and adaptive adjustments are necessary to ensure that the implementation of these techniques does not negatively impact the safety and privacy of adolescents.