The dataset used in this study consists of de-identified data collected from the WA BCYR pilot program [30]. The dataset includes 293 Aboriginal mothers who participated between September 9, 2021, and June 16, 2023, across 6 diverse pilot sites services in metropolitan Perth and regional WA. The BCYR assessment/screen is being offered to all women at pilot sites as part of their routine perinatal care in an additional 30-minute stand-alone appointment [10]. All pregnant women or mothers with infants who accessed participating perinatal services at the pilot sites were eligible to take part in the BCYR assessment. The sampling approach was convenience-based, with the BCYR assessment offered to all eligible Aboriginal mothers attending the pilot sites during the study period.

Ethics approval for this study was obtained from the Human Research Ethics Committee (HREC) - Western Australia Research Governance Service (RGS000000649), Murdoch University (2021/101), and the Western Australian Aboriginal Health Ethics Committee (WAAHEC; HREC553). Access to deidentified data was granted only to participants who provided informed consent. The information sheet, and a consent button, allowing participants to choose whether or not to participate and share their deidentified data for research purposes, were embedded in the digital application.

Observational units were individual patients, with response variables (psychological distress risk) and predictor variables (demographic, social, and behavioral factors). The skipped question’s answer by the participants was assigned a value of –1.

The original dataset contains 345 variables for each participant, covering a wide range of inquiry domains such as strengths and culturally protective factors, common life events, worries, quality of relationships, childhood experiences, family and domestic violence, and tobacco and alcohol and other drug use. Feature selection was performed using the RF to compute variable importance ranking. The algorithm was configured with 500 trees, and the “mtry” parameter was set to the square root of the total number of variables, rounded down to the nearest integer. Initially, the top 30 most significant variables were selected. These variables were then reviewed by the BCYR research team, which included Aboriginal researchers, both Aboriginal and non-Aboriginal health care professionals, and BCYR digital assessment users. Incorporating their domain knowledge and experience, the final list was narrowed down to 20 predictor variables along with the Kessler-5 item psychological distress scale (K5) output variable, for analysis. The final selected variables are listed in Table 1.

The indicator for maternal psychological distress is based on the K5 scale [31]. The K5 scale consists of 5 items, each rated on a 5-point scale from 1 to 5, with all items negatively keyed. The total score ranges from 5 to 25, with a score below 12 indicating low risk (0), and a score of 12 or higher indicating high risk (1) [32]. Five records were excluded from the original dataset due to missing information on the K5 outcome variable. The dataset had a class ratio of 0.65 (low risk) to 0.35 (high risk).

We used 7 ML models to train and evaluate the prediction model on the processed dataset. These models were chosen due to their widespread use and proven effectiveness in health care predictive modeling, particularly with relatively small tabular data [33-35]. The models include random forest (RF) [36], CatBoost (CB) [37], light gradient-boosting machine (LightGBM) [38], extreme gradient boosting (XGBoost) [39], k-nearest neighbor (KNN) [40], and support vector machines (SVM) [41] as blackbox models, and one inherently interpretable glassbox model explainable boosting machines (EBMs) [42]. More descriptions of these selected models are provided in Section I in Multimedia Appendix 1.

We used a 10-fold cross-validation strategy for splitting the training and testing data to ensure a fair model training and evaluation process. Hyperparameters for each model were tuned using grid search with cross-validation. Details for hyperparameter tuning were provided in Section III in Multimedia Appendix 1.

Ten-fold was adopted as it is more suited for relatively small datasets, offering a better balance between bias and variances [43]. We then report the average model performance using a comprehensive set of metrics, including accuracy, precision, recall, F₁-score, and area under the curve (AUC). Precision, recall, F₁-score, and AUC are particularly recognized as robust metrics when dealing with imbalanced class ratios. Additionally, 95% CIs are provided for each performance metric to account for uncertainty.

Different model explanation techniques were used to investigate how factors influence psychological distress and to compare their outputs. First, EBM, as a glassbox model, is inherently explainable and provides both global explanations for the model’s overall behavior, and local explanations for specific predictions on individual cases [42]. We also applied post-hoc explanation techniques, including Shapley additive explanations (SHAP) [44], local interpretable model-agnostic explanations (LIME) [45,46], and partial dependence plots (PDP) [47], to elucidate the predictions of the best-performing blackbox model. SHAP and LIME can offer both global and local explanations. PDP, as a visualization technique, shows how 1 or 2 selected features impact the predicted outcome while keeping all other features constant. More details on these post-hoc explanation techniques are provided in Section II in Multimedia Appendix 1.

Table 2 displays the training and evaluation results of all the ML models across various performance metrics, with the best results highlighted in italics. Among black-box models, RF achieved the highest performance, with an accuracy of 0.829, an F₁-score of 0.736, and an AUC of 0.795. The glass-box model EBM outperformed all models, attaining an accuracy of 0.849, an F₁-score of 0.771, and an AUC of 0.821. Ensemble models, including RF, CB, XGBoost, and LightGBM, demonstrated strong predictive performance, all achieving an accuracy above 0.81. KNN and SVM showed slightly lower accuracy (0.798 and 0.794) and comparable AUC values (0.733 and 0.742). In terms of precision and recall, KNN exhibited the highest precision (0.868) but the lowest recall (0.514), leading to a lower F₁-score (0.621). EBM and RF achieved a better balance, with EBM attaining a precision of 0.829, a recall of 0.727, and an F₁-score of 0.771, while RF reached a precision of 0.820, a recall of 0.680, and an F₁-score of 0.736. Figure 1 plots the receiver operating characteristic curve curves of all the models on one set of testing, showing that EBM and RF achieved the highest AUC values.

The prediction model performance analysis indicates that EBM and RF performed best overall, offering a strong balance between accuracy, F₁-score, and AUC. Ensemble models (RF, CB, XGBoost, and LightGBM) demonstrated strong predictive performance, benefiting from their ensemble-based architecture, which enhances generalization and robustness. While CB, XGBoost, and LightGBM are powerful, they may be more prone to overfitting on small datasets. In contrast, RF uses bagging, which reduces variance and tends to be more robust for small datasets. Among the models, KNN exhibited high precision but low recall, indicating a tendency to miss high-risk cases, which could be a critical limitation for mental health risk detection. In contrast, EBM and RF provided a better balance, making them more suitable for this task.

The global feature importance analysis using EBM, SHAP, and RF identified key predictors of perinatal mental health risk and revealed important interactions between features. The question concerning “Feeling Lonely” consistently emerged as the most influential predictor across models, followed by questions concerning “Blaming Herself,” “Makes Family Proud,” “Life Not Worth Living,” and “Managing Day-to-Day.” Moreover, EBM showed specific multidimensional interactions that add increased weighting to the model’s predictions. For example, “Feeling Lonely” in combination with “Life Not Worth Living,” “Blaming Herself,” or “Partner Angry/Controlling” placed greater predictive power. Similarly, interactions between “Blaming Herself” and “Family Gambles” or “Makes Family Proud” were identified as key joint effects affecting model predictions.

Variations in feature rankings across EBM, SHAP, and RF likely arise from methodological differences in how global feature importance is measured. RF determines importance based on node splits and impurity reduction but does not explicitly capture feature interactions. SHAP estimates feature contributions by considering their marginal effects and interactions across all predictions. EBM computes global importance by aggregating the learned effects of each feature in an additive framework while explicitly modeling interactions through pairwise terms. By using these different global feature importance methods, we can get a more comprehensive understanding of feature importance and uncover key patterns that enhance model interpretability.

Recent advancements in AI-driven predictive models for perinatal mental health have demonstrated varying levels of effectiveness. Previous studies introduced an EBM trained with Aboriginal lived experiences, highlighting the need for culturally sensitive AI applications [27]. Similarly, another study showed that ML models, particularly RF and SVM, could effectively predict psychological distress among Aboriginal perinatal mothers [32]. Other studies have explored the broader application of AI in perinatal health, such as AI’s role in predicting preterm birth and postpartum depression [20,48]. Our study builds upon these findings by integrating XAI techniques and incorporating Aboriginal knowledge and lived experiences to improve transparency in decision-making and support the development of culturally safe AI applications in perinatal mental health.

At the individual level, where responses are highly personal, local explanations from XAI techniques provided case-specific insights, distinguishing between protective and risk factors and illustrating their respective contributions. In combination with global feature importance results, positive family relationships emerged as a key protective factor in mitigating poor perinatal mental health, aligning with findings from Ratajczak [9] and Carlin et al [12]. Similarly, risk factors such as feelings of loneliness and poor partner relationships were consistent with the findings of Carlin et al [12]. This study may offer new ways to identify protective and risk factors in Aboriginal perinatal mental health from an explainable AI-based quantitative perspective and predictive modeling approach. Such models could facilitate the early detection of at-risk individuals and support more personalized, culturally sensitive, strengths-based care.

This study has several limitations that should be acknowledged. First, our model was trained on the dataset obtained through convenience sampling, without a formal sample size calculation. The limited sample size and nonrandom sampling approach may introduce selection bias, potentially limiting the model’s generalizability and increasing the risk of overfitting. While cross-validation techniques were applied to mitigate these risks and assess generalization capability, they cannot fully compensate for the limitations posed by the sampling method and dataset size. Second, the absence of established population parameters prevents direct statistical comparisons with broader populations, making it challenging to assess selection bias and affecting the study’s generalizability. Third, potential biases in assessment responses, such as nonresponse and social desirability bias, may affect data quality and influence model outputs. While XAI techniques provide a way to identify potential distortions, they do not fully quantify these biases, making it difficult to comprehensively assess model fairness and accuracy. Future research should focus on expanding the dataset by incorporating a more diverse and representative sample across different regions, performing external validation using data from different regions, and systematically assessing model fairness. These steps could help enhance the model’s performance, generalizability, and reliability in practice. Fourth, there is a slight imbalance in the outcome class ratio between low risk and high risk, at 0.65 versus 0.35. Given that the class ratio is relatively moderate and ML models especially ensemble techniques like RF can naturally handle this level of imbalance, and class imbalance-robust performance metrics were used for evaluation, no class imbalance techniques were applied. In the future, as the dataset grows and if the class imbalance increases, additional techniques could be implemented to further improve predictive performance. Fifth, the current visual outputs generated by the XAI techniques can be refined through the co-design process to improve their readability and explainability for Aboriginal women and clinicians. Creating a user-friendly, culturally sensitive visual prediction model will ensure that all practitioners can accurately and responsively interpret the results in practice.

We developed and evaluated several ML models powered by XAI techniques to predict perinatal psychological distress in Aboriginal mothers. The explanations provided by different XAI techniques revealed largely consistent patterns of influential protective and risk factors, their interactions, and their impact on prediction outcomes. Continuous collaboration informed by Aboriginal knowledge and lived experience, will further enhance the model. Such a model may have the potential to assist health care professionals in providing more culturally sensitive clinical reasoning, improving holistic assessment interpretations, and reducing unnecessary child protection notifications. Future studies are needed for clinical validation.