Patients diagnosed with breast cancer at the First Affiliated Hospital of Chongqing Medical University from January 2019 to 2023 were comprehensively reviewed. We selected patients who underwent NAC for subsequent analysis. After administering 4-8 cycles of NAC, clinicians, radiologists, and pathologists evaluated the treatment response. A clinical complete response (cCR) was defined as the complete disappearance of all tumor lesions, as confirmed by imaging examination, lasting for a minimum of 4 weeks [14]. A pathological complete response (pCR) was defined as the absence of any residual invasive tumor in both the breasts and axillary lymph nodes (ypT0ypN0) [15]. Patients without a cCR or pCR were enrolled in this study. The adjuvant treatment process was determined according to the guidelines outlined by the Chinese Society of Clinical Oncology (CSCO) and the National Comprehensive Cancer Network (NCCN) [16,17]. Pathological assessment was conducted according to the American Society of Clinical Oncology guidelines (Multimedia Appendix 2) [18-20].

Initially, we established an RSF model using the entire patient cohort. Subsequently, we examined the relationship between clinicopathological variables and survival outcomes using univariate and multivariate Cox regression analyses. We then constructed a Cox regression model using the variables selected based on P<.05 from the multivariate analysis. Furthermore, the Duke University Breast Cancer dataset [21] and data from the Surveillance, Epidemiology, and End Results (SEER) [22] database were used as validation cohorts.

This study was conducted in accordance with the principles of the Declaration of Helsinki and was approved by the Ethics Committee of the First Affiliated Hospital of Chongqing Medical University (ID: 2020–59). Compensation and informed consent were waived due to the retrospective nature of the study and the use of deidentified patient data.

All patients included in this study were interviewed through either outpatient visits or telephone consultations. The follow-up period extended from discharge until April 30, 2024. Disease-free survival (DFS) was selected as the primary metric for assessing the patient survival duration. The DFS was defined as the time from surgery to the occurrence of breast cancer recurrence, the diagnosis of a new primary cancer, or death from any cause, whichever comes first.

The RSF model was constructed using the randomForestSRC package in R (R Foundation for Statistical Computing). A starting model was trained on the entire cohort. The following hyperparameters were configured: number of trees (ntree), node size, mtry, and independent variables. The model underwent 2 key adjustments: (1) hyperparameter tuning via cross-validation to optimize performance and (2) permutation-based variable importance scoring to identify significant predictors. Following optimization, the final model was built. Additionally, variables selected based on P<.05 from the multivariate Cox regression analysis were incorporated into the Cox regression model.

In this study, external validation datasets were obtained from 2 sources: the Duke breast cancer dataset and the SEER dataset. For the Duke dataset, the inclusion criterion was that patients had undergone NAC. The exclusion criteria were (1) any missing or unknown information and (2) patients who achieved a CR to NAC. For the SEER dataset, the inclusion criteria were (1) patients registered in registry 8 and (2) patients diagnosed with a malignant breast tumor. The exclusion criteria were (1) patients with more than 1 primary tumor, (2) any missing information or records marked as “Unknown,” (3) patients who did not undergo NAC, and (4) patients who achieved a CR to NAC. In the Duke dataset, survival time was evaluated as recurrence-free survival (RFS), whereas the SEER dataset used overall survival (OS) as its measure.

The performance of the RSF and the Cox regression model was assessed using the 2 validation cohorts. Key metrics, including the concordance index (C-index), 95% CIs, and the integrated Brier score, were calculated to evaluate the models’ predictive accuracy and calibration. Survival curves were generated using the Kaplan-Meier (K-M) method, stratified by risk group identified using the RSF and the Cox regression model. Additionally, variable importance plots were created to illustrate the contribution of each variable to the models. Finally, the models were validated in the Duke and the SEER dataset.

We further used the RSF model to assess the performance of various molecular subtypes across different datasets, with the results illustrated through time-dependent receiver operating characteristic (ROC) curves. Patients with different subtypes in these datasets were categorized into high- and low-risk groups based on the model’s predictions. The survival differences between these 2 groups were evaluated using K-M curves.

We conducted statistical analysis using RStudio (R version 4.4.2). All independent variables were categorized and presented as absolute counts. To compare categorical data, we performed the chi-square test. The survival time was a continuous variable and was presented as the median (IQR). The Cox proportional hazards model was used to identify prognostic factors through both univariate and multivariate analyses. For the development of the RSF model, we used the randomForestSRC package, while the cph function was used to construct the Cox regression model. The ROC curve and decision curve analysis (DCA) were used to display the performance of the models. Survival analysis was performed using K-M curves, with differences between groups evaluated using the log-rank test. In this study, P<.05 was considered indicative of a statistically significant difference.

The study flowchart and patient inclusion process are illustrated in Figure 1. The internal cohort comprised 306 patients with breast cancer with the following characteristics: the majority (204/306, 66.7%) were aged 40-60 years, 94.8% (290/306) had invasive ductal carcinoma, and receptor status analysis revealed 59.5% (182/306) estrogen receptor (ER)+, 58.5% (179/306) progesterone receptor (PR)–, and 30.7% (94/306) human epidermal growth factor receptor 2 (HER2)+ cases. The most prevalent staging was T2 (185/306, 60.5%), N1 (142/306, 46.4%), and M0 (295/306, 96.4%). Over a median follow-up of 25.9 months (IQR 17.2-36.3), 17.6% (54/306) patients experienced disease-related events. External validation cohorts demonstrated comparable patterns, with the Duke dataset (N=94) showing a similar age distribution (59/94, 63%, aged 40-60 years), HER2-positivity rate (26/94, 28%), and hormone receptor (HR) profiles, while the SEER cohort (N=2760) maintained consistent invasive ductal carcinoma predominance (2506/2760, 90.8%) and staging trends. Full demographic details are presented in Table 1.

Initially, we integrated all independent variables into the RSF model, setting the number of trees (ntree) to 2000. Our findings indicated that the model stabilized when ntree reached 1000. Additionally, variable importance analysis revealed that the variable T stage had significant negative effects on the model’s performance (Multimedia Appendix 3). Consequently, we adjusted ntree to 1000 and included age, histological type, ER, PR, HER2, N stage, and M stage as independent variables to reconstruct the RSF model. Through hyperparameter tuning, we determined that optimal model performance and generalization ability were achieved with a node size of 10 and an mtry of 2 (Multimedia Appendix 4). The ROC curves showed that in the training set, the AUC of the model at 1, 3, and 5 years was 0.811, 0.834, and 0.810, respectively (Figure 2A). The C-index was 0.803 (95% CI 0.747-0.859). The Brier score is shown in Multimedia Appendix 5.

Results of the multivariate Cox regression analysis indicated that age, PR, HER2, N stage, and M stage were significantly associated with survival risks (Table 2). These variables were incorporated into the Cox regression model, which yielded an AUC of 0.763, 0.783, and 0.771 at the 1-, 3-, and 5-year marks, respectively (Figure 2B). The C-index was calculated to be 0.736 (95% CI 0.673-0.799). The Brier score, which was relatively lower than that of the RSF model at each time point, is presented in Multimedia Appendix 5.

A total of 94 patients from the Duke dataset and 2760 patients from the SEER dataset were included in the analysis; the selection process is displayed in Figure 1. For the RSF model, the AUC for the Duke dataset was 0.912 at 1 year, 0.803 at 3 years, and 0.776 at 5 years, as illustrated in Figure 2C, while the AUC values for the SEER dataset were 0.771 at 1 year, 0.729 at 3 years, and 0.701 at 5 years, as shown in Figure 2E. For the Cox regression model, the AUC for the Duke dataset was 0.869 at 1 year, 0.759 at 3 years, and 0.706 at 5 years, with the corresponding ROC curves presented in Figure 2D. For the SEER dataset, the AUC values were 0.823 at 1 year, 0.756 at 3 years, and 0.731 at 5 years, as depicted in Figure 2F.

Multimedia Appendix 6 presents DCA curves for the 2 models at 1-, 3-, and 5-year intervals across all datasets. In both the training set and the Duke dataset, patients derived greater benefits from the RSF model compared to the Cox regression model. Conversely, in the SEER dataset, the benefits for patients using both models were comparable. Furthermore, in the training set, the RSF model identified a predictive cut-off value of 8.70 to categorize patients into high- and low-risk groups (Multimedia Appendix 7). Survival analysis demonstrated that the prognosis for the high-risk group was significantly worse than that for the low-risk group (Multimedia Appendix 8). This cut-off value was also applied to classify patients from both the Duke and SEER datasets into high- and low-risk groups. Survival analysis indicated that patients in the high-risk group had poorer prognoses compared to those in the low-risk group across both datasets (Multimedia Appendix 8). Similarly, the Cox regression model established a predictive cut-off value of 0.27 in the training set to differentiate between high- and low-risk groups (Multimedia Appendix 7). Survival analysis yielded results consistent with those obtained using the RSF model (Multimedia Appendix 8).

We conducted a performance evaluation of the RSF model across various molecular subtypes. The ROC curves indicated that in the internal dataset, the RSF model achieved an AUC of 1.000 for both 1- and 3-year survival rates and 0.748 for the 5-year survival rate for the HR+/HER2+ subtype. For the HR+/HER2– subtype, the AUC values were 0.872 for the 1-year, 0.699 for the 3-year, and 0.778 for the 5-year survival rate. For the HR–/HER2+ subtype, the AUC values were 0.639 for the 1-year, 0.845 for the 3-year, and 0.698 for the 5-year survival rate. For the HR–/HER2– subtype, the AUC values were 0.681 for the 1-year and 0.832 for the 3-year survival rate. Consistent trends were observed in the SEER and Duke validation datasets (Multimedia Appendix 9). The K-M curves further demonstrated the RSF model’s ability to stratify patients into distinct high- and low-risk groups across all 4 molecular subtypes in both internal and SEER datasets. In the Duke dataset, although no statistically significant difference was observed in the HR−/HER2− subtype, low-risk patients still exhibited higher RFS compared to high-risk patients (Multimedia Appendix 10).

In this study, our key findings demonstrated that the RSF model outperforms Cox regression in predicting survival risk for nonresponders post-NAC, with validated generalizability across external cohorts. The RSF model also demonstrated consistent effectiveness when analyzing various molecular subtypes. The findings highlight the potential of machine learning techniques, particularly the RSF, in enhancing prognostic accuracy and guiding clinical decision-making in oncology.

Previous research has demonstrated that achieving a pCR following NAC is associated with significantly improved event-free survival and OS [3]. Consequently, numerous studies have concentrated on predicting tumor responses to NAC. For instance, Zhao et al [23] constructed machine learning models to predict the pCR to NAC based on clinicopathological variables. Similarly, Zhang and coworkers [24-30] developed machine learning models that incorporated clinicopathological features, radiomic features, and pathomic features to forecast the pCR following NAC. Additionally, Sammut et al [31] and Chen et al [32] created models using multi-omics data. These studies have highlighted the substantial value of machine learning models in predicting patients’ responses to NAC.

However, as previous research has confirmed that patients with no pCR tend to have a worse prognosis and a higher risk for adverse events, it is important to note that few studies have explored risk stratification among those patients. In one of our earlier studies, we tried to develop a random forest model to predict event occurrence among patients with breast cancer with no response to NAC [33]. At the same time, we aimed to create a model that directly predicted patients’ survival risk. The RSF algorithm integrates random forests with survival analysis, enabling the prediction of individual event probabilities and survival time. Compared to the traditional Cox regression, the RSF model offers several advantages: It is not limited by the proportional hazards assumption, it can effectively handle high-dimensional data, and it demonstrates strong generalization capabilities [11]. Liao et al [9] developed a prediction model using 10 machine learning algorithms across 101 combinations to forecast cancer-related mortality in patients with gastric neuroendocrine neoplasms. They finally found that the RSF model obtains the highest AUC value [9]. Similar results have been found in other studies [8,10-13]. In our study, we also found that the RSF model outperforms the Cox regression model, a finding that was further validated using the Duke dataset.

One of the strengths of our study is the robust validation of the RSF model using 2 independent validation cohorts. The C-index demonstrated the model’s predictive accuracy, suggesting that it can reliably stratify patients based on their survival risk. Additionally, the survival time metrics varied across datasets: the training set used DFS, whereas the Duke dataset was based on the RFS, and the SEER dataset used OS. The DFS status encompassed disease recurrence, new primary diseases, and death. Consequently, the model demonstrated strong adaptability across datasets that use these endpoints as status indicators. Therefore, we suggested using DFS to construct time-to-event predictive models in future studies. Another issue pertains to the selection of independent variables. Initially, we tried to develop the RSF model using variables selected through multivariate analysis in the Cox regression; however, the results were not satisfactory. Subsequently, we used the variable importance metrics from the RSF algorithm to identify variables that were ultimately included in the model. Hence, we also suggested using a variable selection method that aligns with the specific model being used.

To the best of our knowledge, our study demonstrated that a more complex algorithm could effectively predict the survival risk of patients with breast cancer without a CR post-NAC, based solely on clinicopathological variables. Nevertheless, there are limitations of our study. First, the sample of the training set was relative small, which might affect variable selection and the parameters set. Second, the follow-up duration in our study was relatively short, with only a limited number of patients followed for more than 5 years. This might have constrained the models’ ability to accurately predict long-term prognosis. Third, the retrospective nature of the data collection might introduce biases, and the findings should be interpreted with caution. Fourth, we included only a limited subset of clinicopathological variables, and further exploration is needed to assess the potential inclusion of additional variables. Additionally, we noted that the performance of the 2 models was comparable in the SEER dataset, with the Cox regression model performing slightly better. This might be attributed to the limited training sample size, which could restrict the performance of the RSF model. Nevertheless, our findings confirm the potential of the RSF model for predicting prognosis using clinicopathological variables in patients. Further prospective study is necessary to confirm its applicability. Moreover, the RSF model is particularly well suited for handling higher-dimensional variables. Thus, future studies should also explore the integration of radiomics, pathomics, and molecular data to enhance the predictive power of the model [34-36].

Our study highlighted the feasibility and effectiveness of using an RSF model based exclusively on clinicopathological variables to predict survival risk in patients with breast cancer who do not achieve a CR after NAC. This approach could enhance clinical practice by assisting oncologists in identifying high-risk patients who might benefit from more aggressive treatment strategies or closer monitoring. As we continue to refine and validate this model, we anticipate that it would significantly contribute to the advancement of personalized medicine in breast cancer care.