We searched the PubMed, Web of Science, Embase, and Cochrane databases comprehensively from their inception to July 24, 2022. A combination of keywords and subject headings was used for the search, and the language was restricted to English. The main search terms included stroke, cerebrovascular accident, apoplexy, machine learning, onset, and stroke time. The specific search strategy is presented in Multimedia Appendix 1. Two researchers independently searched the literature, and any disagreements were solved by a third researcher. We performed a further, manual search of the reference lists of the included articles to find potential eligible studies.

After removing duplicates using EndNote (Thomson ResearchSoft), we screened the literature according to the inclusion and exclusion criteria presented in Textbox 1.

The retrieved documents were imported into EndNote X9 for management. The titles and abstracts were screened to exclude irrelevant studies. The full texts of the remaining articles were downloaded and read to select eligible studies. Before data extraction, we prepared a standard electronic information extraction form. The primary outcomes included in the study are C index, sensitivity, and specificity. The abovementioned literature screening and information extraction were independently carried out by 2 researchers, and any disagreement was resolved by a third researcher.

The PROBAST (Prediction Model Risk of Bias Assessment Tool) was used [22] to evaluate the risk of bias in the included studies. This tool contains several questions across 4 different domains: participants, predictors, outcome, and analysis. The overall applicability of the included studies was also assessed.

The 4 domains comprise 2, 3, 6, and 9 specific questions, respectively, and each question was answered as “yes,” “probably yes,” “no,” “probably no,” or “no information.” If a domain included at least 1 response of “no” or “probably no,” it was considered at high risk. A domain with all the responses being “yes” or “probably yes” was judged low risk. When all domains were considered low risk, the overall risk of bias was rated low. When at least 1 domain was considered high risk, the overall risk of bias was high. Two researchers independently carried out the risk of bias assessment using PROBAST and cross-checked their results. If there was any dispute, a third researcher was consulted to assist in the final decision.

We conducted a meta-analysis of the evaluation indicators (C index and accuracy) of the machine learning models. If the C index had no 95% CI or SE, we estimated the SE with the calculation method proposed by Debray et al [23]. Given different variables and inconsistent parameters in learning models, a random effects model was preferred in the meta-analysis of the C index. In addition, we used a bivariate mixed effects model to perform the meta-analysis of sensitivity and specificity. The meta-analysis of our study was conducted using R (version 4.2.0, R Development Core Team).

After removing the duplicates, a total of 843 papers were left. Based on the title and abstract screening, the full texts of 17 papers were downloaded and read. Finally, 13 eligible papers were included in this study. The details of the literature screening process are shown in Figure 1.

There were 11 case-control studies and 2 cohort studies. Among the included studies [17,18,24-34], 5 were from China, 3 from Belgium, 4 from the United States, and 1 from South Korea (shown in Table 1) There were 55 models involved, with 41 models in the training set and 14 in the validation set. These models were mostly based on logistic regression (LR), with radiomics-based modeling variables.

All 41 models in the training set were radiomics-based machine learning models. The inclusion and exclusion criteria for the cases were relatively reasonable. In addition, the onset time records were suitably reliable. In terms of data generation, only one study described the processing method for missing clinical values. Of note, no missing values in the radiomics data were mentioned. In the modeling process, the verification set was mostly generated through random sampling for internal verification, while 3 multicenter studies had independent external verification sets. There were fewer than 100 samples in the verification set. The risk of bias assessment of the included studies is presented in Figure 2.

Subgroup analyses of the C index were performed for onset times within 3 hours, 4.5 hours, and 6 hours. The analysis of onset time within 3 hours showed that the C index was 0.90 (95% CI 0.81-0.99) in the training set. For the onset time within 4.5 hours, there were 38 models in the training set and 13 models in the validation set. The C index for 4.5 hours was 0.80 (95% CI 0.77-0.83) in the training set and 0.79 (95% CI 0.71-0.86) in the validation set. Regarding the time within 6 hours, there were 2 models in the training set and 1 in the validation set. The C index for 6 hours was 0.76 (95% CI 0.71-0.81) in the training set and 0.73 (95% CI 0.62-0.84) in the validation set. The C index in the training set is depicted in Figure 5, and the C index of the validation set is presented in Figure 6.

Our analysis suggests that machine learning showed ideal performance in predicting and identifying the time of symptom onset in stroke patients. Accurate prediction of stroke onset time is extremely important for identifying the time window of thrombolytic therapy for stroke patients [18]. However, stroke patients usually do not know the specific time of symptom onset. Our research results can provide a reference for the clinical prediction of stroke onset time. We included 10 main types of models, including LR, support vector machine (SVM), artificial neural network (ANN), boosting, deep learning (DL), random forest (RF), decision tree (DT), generalized linear model (GLM), k-nearest neighbor (KNN), and naive Bayes (NB). Generally, the models showed ideal predictive performance, with an overall C index of 0.800 (95% CI 0.773-0.826) in the training set and 0.781 (95% CI 0.709-0.852) in the validation set. The overall C index of the LR model for predicting the stroke onset time was relatively high, at 0.807 (95% CI 0.741-0.872) in the training set and 0.807 (95% CI 0.691-0.923) in the validation set, despite high heterogeneity across the studies. These results indicate that the performance of machine learning in identifying stroke onset time is excellent, and it can be used as a potential tool to determine when a stroke begins. Furthermore, the sensitivity and specificity were 0.76 (95% CI 0.73-0.80) and 0.79 (95% CI 0.74-0.82) in the training set, and 0.81 (95% CI 0.68-0.90) and 0.83 (95% CI 0.73-0.89) in the validation set.

Machine learning models have been widely used in the health care industry and show favorable performance in the stratification of patients. Numerous studies have been carried out on the use of machine learning models to predict the risk of stroke, the prognosis of acute stroke, and motor function recovery in stroke patients after treatment, as well as for the early identification of ischemic stroke patients at a high risk of recurrence [21,35-37]. Despite many publications on outcomes and treatment response in stroke patients, the use of MRI to determine the time of symptom onset is rarely researched. Therefore, this study explored the value of applying machine learning in predicting stroke onset time.

In our study, the variable included in the model for predicting the time of stroke onset was MRI, specifically diffusion-weighted imaging (DWI) and fluid-attenuated inversion recovery (FLAIR). A mismatch between DWI and FLAIR has been adopted to identify stroke patients who are likely to benefit from thrombolysis therapy [38,39]. Previous studies have reported that DWI-FLAIR mismatch could achieve a specificity of 0.60 to 0.80 and a moderate sensitivity of 0.5 to 0.6 [40-42]. Machine learning is a branch of artificial intelligence that can use a variety of image features, including those invisible to humans, with a favorable predictive accuracy [16,18,43].

In addition, our research shows that the LR model had the best predictive performance among the included models. Moreover, some institutions that specialize in detecting tumors have constructed LR-based machine learning models, which have achieved good results [44-46]. Based on existing research and our results, we vigorously recommend promoting the development of nomograms based on LR models for research on multiple diseases.

This study is the first meta-analysis to investigate the value of applying machine learning in predicting the time of symptom onset in stroke patients, comprehensively promoting the development of accurate treatment of stroke. At the same time, it will be useful for identifying stroke onset time once its feasibility is confirmed. However, there are still some limitations to our review. First, the included studies are mainly single-center, and it is difficult to eliminate the impact of socioeconomic background. Second, although we have conducted a comprehensive search, the number of included studies is still small. Third, the diversity of the included models resulted in heterogeneity. Since it is difficult to avoid heterogeneity in machine learning models, more large-scale, multicenter studies are desired to reduce heterogeneity.

Machine learning has ideal performance in identifying stroke onset time, and it can be used as a potential tool to determine when a stroke begins. Nonetheless, its predictive accuracy still needs to be improved. As a result, minimally invasive or noninvasive, easily collected, and efficient predictors should be considered in future research to improve the predictive accuracy of machine learning. Moreover, it is necessary to explore reasonable image-segmentation and texture-extraction methods in terms of radiomics. In addition, comprehensive research on the popularization of machine learning and the value of its application is desired. Furthermore, the promotion and application of machine learning should take into account differences in ethnic backgrounds.