Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Wednesday, July 01, 2020 at 8:00 PM to 10:00 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?


Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 16.09.20 in Vol 22, No 9 (2020): September

Preprints (earlier versions) of this paper are available at, first published May 28, 2020.

This paper is in the following e-collection/theme issue:

    Original Paper

    Automatic Grading of Stroke Symptoms for Rapid Assessment Using Optimized Machine Learning and 4-Limb Kinematics: Clinical Validation Study

    1Cerebro-Cardiovascular Disease Research Center, Yonsei University College of Medicine, Seoul, Republic of Korea

    2Department of Radiology, Yonsei University College of Medicine, Seoul, Republic of Korea

    3Health-IT Center, Yonsei University College of Medicine, Seoul, Republic of Korea

    4Department of Neurology, Yonsei University College of Medicine, Seoul, Republic of Korea

    Corresponding Author:

    Hyo Suk Nam, MD, PhD

    Department of Neurology

    Yonsei University College of Medicine

    50 Yonsei-ro, Seodaemoon-gu

    Seoul, 03722

    Republic of Korea

    Phone: 82 222280245



    Background: Subtle abnormal motor signs are indications of serious neurological diseases. Although neurological deficits require fast initiation of treatment in a restricted time, it is difficult for nonspecialists to detect and objectively assess the symptoms. In the clinical environment, diagnoses and decisions are based on clinical grading methods, including the National Institutes of Health Stroke Scale (NIHSS) score or the Medical Research Council (MRC) score, which have been used to measure motor weakness. Objective grading in various environments is necessitated for consistent agreement among patients, caregivers, paramedics, and medical staff to facilitate rapid diagnoses and dispatches to appropriate medical centers.

    Objective: In this study, we aimed to develop an autonomous grading system for stroke patients. We investigated the feasibility of our new system to assess motor weakness and grade NIHSS and MRC scores of 4 limbs, similar to the clinical examinations performed by medical staff.

    Methods: We implemented an automatic grading system composed of a measuring unit with wearable sensors and a grading unit with optimized machine learning. Inertial sensors were attached to measure subtle weaknesses caused by paralysis of upper and lower limbs. We collected 60 instances of data with kinematic features of motor disorders from neurological examination and demographic information of stroke patients with NIHSS 0 or 1 and MRC 7, 8, or 9 grades in a stroke unit. Training data with 240 instances were generated using a synthetic minority oversampling technique to complement the imbalanced number of data between classes and low number of training data. We trained 2 representative machine learning algorithms, an ensemble and a support vector machine (SVM), to implement auto-NIHSS and auto-MRC grading. The optimized algorithms performed a 5-fold cross-validation and were searched by Bayes optimization in 30 trials. The trained model was tested with the 60 original hold-out instances for performance evaluation in accuracy, sensitivity, specificity, and area under the receiver operating characteristics curve (AUC).

    Results: The proposed system can grade NIHSS scores with an accuracy of 83.3% and an AUC of 0.912 using an optimized ensemble algorithm, and it can grade with an accuracy of 80.0% and an AUC of 0.860 using an optimized SVM algorithm. The auto-MRC grading achieved an accuracy of 76.7% and a mean AUC of 0.870 in SVM classification and an accuracy of 78.3% and a mean AUC of 0.877 in ensemble classification.

    Conclusions: The automatic grading system quantifies proximal weakness in real time and assesses symptoms through automatic grading. The pilot outcomes demonstrated the feasibility of remote monitoring of motor weakness caused by stroke. The system can facilitate consistent grading with instant assessment and expedite dispatches to appropriate hospitals and treatment initiation by sharing auto-MRC and auto-NIHSS scores between prehospital and hospital responses as an objective observation.

    J Med Internet Res 2020;22(9):e20641




    Motor weakness is a typical manifestation in various neurological disorders, including stroke, spinal cord injury, and traumatic brain injury. In addition, it is a major obstacle to functional recovery after the treatment of those diseases. As an example of motor weakness, unintentional drift is an indication of arm weakness; this is mainly caused by subtle damages in the motor pathway from the brain to the spinal cord [1]. If the supinator muscles in the upper limb are weaker than the pronator muscles in the presence of upper motor neuron lesion, the arm drifts downward and the palm turns toward the floor. The pathological response is for one of the arms to drift (up, down, or out). Therefore, motor weakness is a major sign in the FAST (face drooping, arm weakness, speech slurring, and time to call) protocol for stroke patients [2].

    Rapid detection of such motor weakness is critical because acute treatments, including thrombolysis or thrombectomy, are performed in a constrained time window. More importantly, diagnosis can be established through bedside examination by specialists because it is a qualitative measurement. If the symptom occurs outside a hospital, a substantial time delay can cause poor outcomes for acute stroke patients [3-5]. In addition, the objective and accurate neurological assessments are not possible by mere visual examination because the examiner cannot easily trace the movement using the conventional neurological examination when there are subtle weaknesses. Therefore, systems need to automatically detect motor deficits using sensor data in real time.

    However, operating such systems in a real environment requires a significant effort in integrating new systems into an emergency protocol. This is because interruptions caused by the attachment of sensors on patients’ bodies and the initiation of the recording process can affect the streamlined structure of emergency protocols. However, evaluation methods are still required to identify stroke patients, as they can be instantly used in the communication among patients or caregivers, emergency call centers, and hospitals. In addition to a sensor-based measurement tool that was demonstrated useful in detecting subtle motor weakness in our previous study [6], the grading of stroke severity can be informed remotely and used in the emergency medical service (EMS) and hospital system.

    In the field and in clinical environments, various grading methods exist for identifying ischemic stroke patients with motor weakness [7-10]. The National Institutes of Health Stroke Scale (NIHSS) score [11,12] and Medical Research Council (MRC) score [13,14] have been used as typical assessment indicators for stroke in the clinical environment. The rapid arterial occlusion evaluation scale, the Cincinnati stroke triage assessment tool, and the prehospital acute stroke severity scale are grading methods in the field environment. In this study, we implemented auto-NIHSS and auto-MRC systems to grade the NIHSS and modified MRC scores to assess patients in the clinical environment. We used subdivided MRC scores (10-grade MRC) instead of a 6-grade MRC to define subtle differences, as shown in Table 1.

    Table 1. NIHSS and MRC grades for muscle power assessment.
    View this table


    Participants and Data

    A total of 17 participants were recruited; 15 participants (10 male and 5 female participants) were finally enrolled and completed 4-limb drift test trials. To estimate the scores of patients with severity, we performed the assessment shortly after admission to a stroke unit. The ages of the participants ranged from 44 to 92 years, with a mean of 68.6 years (SD 16.11). Exclusion criteria were patients (1) who had a substantial weakness that prevented arm or leg raising against gravity, (2) who were not able to sit and who had bilateral arm weakness or preexisting chronic arm weakness, and (3) who had aphasia, neglect, peripheral neuropathy, myopathy, or joint deformity. This study was approved by the Severance Hospital Institutional Review Board, and informed consent was obtained from all participants.

    Figure 1 shows patient enrollment and data preparation for auto-NIHSS and auto-MRC grading. Description of data composition for training, validation and testing is detailed in the section on system design.

    Figure 1. Patient enrollment and data set for automatic grading system. MRC: Medical Research Council; NIHSS: National Institutes of Health Stroke Scale; SMOTE: synthetic minority oversampling technique.
    View this figure

    System Design

    The entire process of the system is shown in Figure 2. The system is composed of 2 parts, the measurement and the grading units. The measurement unit sets up sensors and Bluetooth connection with the primary information of patients.

    We measured the upper left and upper right limb movements using sensors on both wrists of patients, who were asked to stretch and hold their arms for 20 seconds, as shown in Figure 3. For the lower left and lower right limb drift tests, patients were asked to lift and stretch their left or right leg for 20 seconds.

    Figure 2. Automatic grading process. MRC: Medical Research Council; NIHSS: National Institutes of Health Stroke Scale.
    View this figure
    Figure 3. Schematic of upper and lower limb sensors and corresponding segment axes.
    View this figure

    The pseudo-code of the measurement unit is shown in Multimedia Appendix 1. For each time frame i, the rotational transformation from the limb into the reference frame xyz is denoted as . The corresponding rotation matrices R for each angle are defined using the of the accelerometer signals for the ith frame. Subsequently, the degree of drift, θdrift, is calculated and used in key features of machine learning classification.

    After collecting the series of 4-limb movements during the test time, the grading unit analyzes the kinematic features. Subsequently, the machine learning algorithm is trained to estimate the NIHSS and MRC scores of each limb. Algorithm 2 (in Multimedia Appendix 2) shows the process of feature extraction, data generation, and model training for the optimized classification of auto-NIHSS and auto-MRC.

    In the feature extraction process, features as predictors of limb paralysis were extracted using a series of measured data. In this study, the duration of the drift test (ttest) was set to 20 seconds; however, analysis started 10 seconds after the examination started (tstart) to exclude the initial dip. The average, maximum, and oscillation of drift caused by paralysis for each limb and demographic features were fed to train the machine learning algorithms.

    In the data generation process, we adopted the synthetic minority oversampling technique (SMOTE) [15], leveraging the K-nearest neighbor (K-NN), to solve the imbalanced problem that is typical in machine learning studies in medicine [16-18]. The SMOTE with K-NN generated ng samples for each grade. Therefore, ngc records were used to construct a grading model with c classes. In this study, ng was set to 120 for auto-NIHSS (c=2) and 80 for auto-MRC (c=3) to compose the training data with 240 (ttrain) instances. Apart from the training data, the original data set with 60 records remained for the test data, as shown in Figure 1.

    In the training process, 5-fold cross-validation was applied to reduce overfitting and generalize the model [19]. In the optimization process, the fitted support vector machine (SVM), as well as ensemble models among various SVM kernels and boosting algorithms with tuned hyperparameters, were searched via Bayes optimization in 30 trials for each model [20]. The grading models were implemented and evaluated in MATLAB R2020a (MathWorks Inc) [21].


    Sensor Data Characteristics

    The system measured the drift of 4 limbs and extracted the kinematic features, as shown in Multimedia Appendix 3. The characteristics of the patients and test data are summarized in Table 2. The grade distribution of clinical scores was not regularized between limbs, as shown in Figure 4. For example, the upper left MRC group had 10 patients graded as MRC 9, 2 patients graded as MRC 8, and 3 patients graded as MRC 7. Among 13 MRC 8 instances, 7 were evaluated as NIHSS 1, whereas 6 were evaluated NIHSS 0. We constructed auto-MRC, which discriminated instances of grades with a data ratio of 13:13:34, whereas auto-NIHSS performed binary classification with a data ratio of 40:20.

    Figure 4. Grade distribution of NIHSS and MRC. MRC: Medical Research Council; NIHSS: National Institutes of Health Stroke Scale.
    View this figure
    Table 2. Summary of patients and test data.
    View this table

    Evaluation Outcomes

    We evaluated the performance of the system in terms of the accuracy, sensitivity, specificity, precision, F1 score, and area under the receiver operating characteristics curve (AUC) with a confusion matrix.

    The statistical plots in Figure 5 show the patterns of the average, maximum, and oscillation of the 4-limb features of each NIHSS grade. Auto-NIHSS discriminated those features, as shown in the confusion matrices in Figure 6. The result shows that the proposed autonomous grading achieved an accuracy of at least 80% and that the overall accuracy was 81.7%, as shown in the summary of performance in Table 3. The AUC of auto-NIHSS reached 0.912, as depicted in the receiver operating characteristics curves in Figure 6. The sensitivity of the NIHSS grading reached 0.825 with the SVM and 0.875 with the ensemble. The specificity was 0.750 for both models.

    Auto-MRC discriminates instances into 3 MRC grades, and the statistical plots of movement features are depicted in Figure 7. The mean AUC was 0.870 for the SVM and 0.877 for the ensemble, as shown in Figure 8. Table 4 shows the summarized performance of auto-MRC; the average accuracy, sensitivity, and specificity for the MRC grading were 0.775, 0.717, and 0.876, respectively.

    Figure 5. Statistical plots of 4-limb features of NIHSS grades. NIHSS: National Institutes of Health Stroke Scale.
    View this figure
    Figure 6. Confusion matrix and receiver operating characteristic of auto-NIHSS grading using (A) support vector machine and (B) ensemble learning. AUC: area under the receiver operating characteristics curve; NIHSS: National Institutes of Health Stroke Scale.
    View this figure
    Figure 7. Statistical plots of 4-limb features of MRC grades. MRC: Medical Research Council.
    View this figure
    Figure 8. Confusion matrix and receiver operating characteristic of auto-MRC grading using (A) support vector machine and (B) ensemble learning. AUC: area under the receiver operating characteristics curve; MRC: Medical Research Council.
    View this figure
    Table 3. Performance of auto-NIHSS grading.
    View this table
    Table 4. Performance of auto-MRC grading.
    View this table


    Importance of Objective and Fast Assessment of Stroke Severity

    The notion “time is brain” is valid in treating stroke patients. Intravenous tissue plasminogen activator (IV tPA) within 4.5 hours of stroke onset is the only therapy for acute ischemic stroke [22]. Subsequently, endovascular thrombectomy (EVT) has been a standard of care for patients with acute ischemic stroke caused by large artery occlusion within 6 to 24 hours of onset, based on successful large randomized clinical trials [23]. Reperfusion therapy, including IV tPA and EVT, for acute ischemic stroke is time sensitive (ie, an earlier treatment yields a better outcome). As the onset-to-intervention time is composed of prehospital and in-hospital phases, patients who arrive early have more chances of appropriate treatment [24-27]. Delays in hospital admission and the preparation before treatment affect the prognosis of patients [28]. In Goyal et al [24], the authors reported that the most significant issue was getting the correct patient to the correct hospital quickly. In Sukumaran et al [27], strategies for stroke patient workflow optimization were suggested by analyzing and solving prehospital and preprocedural bottlenecks. The interhospital transfer is directly associated with delays in onset to reperfusion time, which results in the poor outcome of stroke patients; therefore, the timely triage of patients is a significant bottleneck [27].

    The importance of accurate and objective assessments of stroke severity in telemedicine and telestroke strategies has been discussed in numerous studies [29]. In particular, the timing constraint in performing reperfusion therapy, which has been shown to significantly reduce mortality, invokes the development of efficient systems and protocols in prehospital care or emergency medical systems. Researchers have addressed the fact that the rapid and accurate evaluation of stroke severity can aid in identifying patients for treatments and accelerate an urgent streamlined process. In the study by Andsberg et al [30], a prehospital ambulance stroke test was performed to score the severity of stroke through commands, answers, and observations. The remote assessment of stroke using smartphones was proposed and compared with bedside examination in calculating the NIHSS score [31]. However, most assessments in those systems used conservative observation or campaigns that were subjective and unreliable between testers. Modern communication, sensor technology, and machine learning can solve this problem through accurate measurements and the fast determination of assessment in a prehospital or remote environment [29,32,33]. A previous study evaluated arm function in activities using kinematic exposure variation analysis and inertial sensors [34]. A mobile-based walk test was developed to report patients’ walking ability [35], and upper limb impairments in stroke patients were measured using inertial sensors in the home environment [33]. Such sensor-based testing enables objective evaluation regardless of the testers or place.

    Utility of Consistent Grading Method as an Agreement Between Prehospital and Hospital Environment

    The necessity of a controlled test is revealed in the results of previous studies for monitoring daily living. Motor recovery was monitored using accelerometers, and the NIHSS motor index was estimated in the study by Gubbi et al [36]. However, the movement in daily living limited the accuracy of estimation to 56% for the low index. Activity monitoring in most sensor-based studies involved trials that were not approved by clinical protocols. Those systems limited extensibility as a standard of remote monitoring systems, although they were efficient in tracking the progress or the treatment outcome.

    In addition to rapid and accurate measurements, we aimed to increase the utility of the assessment system in the prehospital and hospital environments. At every phase of the prehospital process, consistent methods to conduct assessments can reduce errors and delays in communication among the participants of a community’s emergency group. Therefore, automatic scoring can facilitate agreement in assessments among patients, caregivers, paramedics, and medical staff. With regard to bottleneck analysis in acute stroke treatment, the rapid identification of neurological deficits and assessment of motor grading will aid EMS personnel in transporting patients to a comprehensive stroke center because hospitals may be limited in terms of stroke unit availability and resources. In Berglund et al [26], the importance of stroke identification without meeting the patient or without neurological examination was asserted; the time to treatment can be decreased with the high-priority dispatch of ambulances through early identification of stroke from emergency calls. In the hyper acute stroke alarm study [25], researchers observed that higher prehospital priority levels of stroke improved thrombolysis frequency and time to stroke unit. The stroke identification by EMS dispatchers during emergency calls varied between 31% and 57%, as identifying stroke can be a challenge without examination [26].

    Therefore, we developed an automatic grading system, leveraging multiclassification of machine learning using typically performed tests and grading in clinics. Our proposed solution uses controlled observations of drift tests in clinics and can estimate the assessment by neurologists. Therefore, the scores by the automatic grading system can be instantly used for communication in an objective manner.

    Data and Techniques for Clinical Scoring by Machine Learning

    A considerable number of studies have used artificial intelligence, including machine learning, to estimate clinical scores and assess patients or provide warnings regarding adverse events [37-40]. In those studies, a series of various techniques were used according to the scale of scores, the capacity of collected data, and the skewness of data. Following the significant development of enhanced algorithms, data with significant meaning have gained importance. However, as addressed in Li et al [41], real-world data have a long-tail pattern with a significant imbalance in quality and quantity. Many algorithms have used public big data to develop new algorithms and build models; however, real-world applications have completely different data quality and quantity and cannot directly apply those models. This situation is particularly severe in medicine, as discussed in Hulsen et al [42]. The availability of qualified data differs by disease, severity of disease in patients, and difficulty of collection [43]. Big data from electronic medical records that are already facilitated in hospital information systems can be used in comparatively easy tasks for medical artificial intelligence. The recent success of medical artificial intelligence requires significant effort and cost in collecting and labeling data [44,45]. In addition, machine learning for sporadic events in emergencies or patients with rare diseases is affected by data deficiency. This is because interventions for collecting data can affect the prognosis of treatment due to the possible delay in the rapid streamlining of treatment processes. Previous feasibility studies have stated that the difficulty in real-time capturing of acute neurological disorders was the main limitation in the research [33,46].

    The learning models with imbalanced data were affected by low precision or recall in the validation and test phases, although they achieved high accuracy for a large number of data in the majority groups [47]. Recently, techniques to solve this data skewness, including data augmentation, transfer learning, and deep imbalanced learning, were emphasized [48-51]. Studies on deep learning that extract filtered features derived from raw data have attempted to solve the problem by knowledge transfer from pretrained models [52,53] or with data augmentation [54,55]. Machine learning with records can cope with the imbalance problem through sampling, cost-sensitive learning, boosting algorithms, and skew-related performance metrics [47,56]. We used the SMOTE to balance between classes in the training phase and applied techniques, including RUSBoost, in optimized ensemble machine learning. To compare different models according to their precision on each class, the F measure is typically used as a performance metric [57]; additionally, we validated the performance of the proposed solution using the AUC and F1 scores. Consequently, the performances of auto-NIHSS and auto-MRC indicated the acceptable AUC, sensitivity, specificity, and F1 score as real-world applications with data skewness.


    Accurate monitoring and grading of motor weakness are critical for the appropriate assessment of stroke severity, particularly for reliable and consistent evaluations. We developed an automatic grading system to assess proximal motor weakness using the kinematic features of unintended drift of 4 limbs. We trained optimized machine learning models and obtained promising results in scoring NIHSS and MRC. The objective scoring of neurological deficits can be used to identify stroke patients, dispatch patients to the appropriate medical center, and expedite treatment preparation.


    This research was supported by a grant funded by the Ministry of Science and ICT (NRF-2020R1A2C1013152) and by a grant of the Korea Health Technology Research and Development Project funded by the Ministry of Health and Welfare (HI19C0481, HC19C0028), Republic of Korea.

    Conflicts of Interest

    None declared.

    Multimedia Appendix 1

    Algorithm of measurement unit for automatic grading.

    PNG File , 699 KB

    Multimedia Appendix 2

    Algorithm of the grading unit for extracting features and training machine learning algorithms. MRC: Medical Research Council; NIHSS: National Institutes of Health Stroke Scale; SMOTE: synthetic minority oversampling technique.

    PNG File , 1029 KB

    Multimedia Appendix 3

    Sample measurement of unintended drift of limbs.

    PNG File , 1259 KB


    1. Darcy P, Moughty AM. Pronator Drift. N Engl J Med 2013 Oct 17;369(16):e20. [CrossRef]
    2. Nor AM, McAllister C, Louw S, Dyker A, Davis M, Jenkinson D, et al. Agreement Between Ambulance Paramedic- and Physician-Recorded Neurological Signs With Face Arm Speech Test (FAST) in Acute Stroke Patients. Stroke 2004 Jun;35(6):1355-1359. [CrossRef]
    3. Kim J, Fonarow GC, Smith EE, Reeves MJ, Navalkele DD, Grotta JC, et al. Treatment With Tissue Plasminogen Activator in the Golden Hour and the Shape of the 4.5-Hour Time-Benefit Curve in the National United States Get With The Guidelines-Stroke Population. Circulation 2017 Jan 10;135(2):128-139. [CrossRef]
    4. Kodankandath TV, Wright P, Power PM, De Geronimo M, Libman RB, Kwiatkowski T, et al. Improving Transfer Times for Acute Ischemic Stroke Patients to a Comprehensive Stroke Center. J Stroke Cerebrovasc Dis 2017 Jan;26(1):192-195. [CrossRef] [Medline]
    5. Threlkeld ZD, Kozak B, McCoy D, Cole S, Martin C, Singh V. Collaborative Interventions Reduce Time-to-Thrombolysis for Acute Ischemic Stroke in a Public Safety Net Hospital. J Stroke Cerebrovasc Dis 2017 Jul;26(7):1500-1505. [CrossRef] [Medline]
    6. Park E, Chang H, Nam HS. Use of Machine Learning Classifiers and Sensor Data to Detect Neurological Deficit in Stroke Patients. J Med Internet Res 2017 Apr 18;19(4):e120 [FREE Full text] [CrossRef] [Medline]
    7. Singer OC, Dvorak F, du Mesnil de Rochemont R, Lanfermann H, Sitzer M, Neumann-Haefelin T. A Simple 3-Item Stroke Scale. Stroke 2005 Apr;36(4):773-776. [CrossRef]
    8. Pérez de la Ossa N, Carrera D, Gorchs M, Querol M, Millán M, Gomis M, et al. Design and Validation of a Prehospital Stroke Scale to Predict Large Arterial Occlusion. Stroke 2014 Jan;45(1):87-91. [CrossRef]
    9. Katz BS, McMullan JT, Sucharew H, Adeoye O, Broderick JP. Design and Validation of a Prehospital Scale to Predict Stroke Severity. Stroke 2015 Jun;46(6):1508-1512. [CrossRef]
    10. Hastrup S, Damgaard D, Johnsen SP, Andersen G. Prehospital Acute Stroke Severity Scale to Predict Large Artery Occlusion. Stroke 2016 Jul;47(7):1772-1776. [CrossRef]
    11. Heldner MR, Jung S, Zubler C, Mordasini P, Weck A, Mono M, et al. Outcome of patients with occlusions of the internal carotid artery or the main stem of the middle cerebral artery with NIHSS score of less than 5: comparison between thrombolysed and non-thrombolysed patients. J Neurol Neurosurg Psychiatry 2015 Jul;86(7):755-760. [CrossRef] [Medline]
    12. Williams LS, Yilmaz EY, Lopez-Yunez AM. Retrospective assessment of initial stroke severity with the NIH Stroke Scale. Stroke 2000 May;31(4):858-862. [CrossRef] [Medline]
    13. Bestall JC, Paul EA, Garrod R, Garnham R, Jones PW, Wedzicha JA. Usefulness of the Medical Research Council (MRC) dyspnoea scale as a measure of disability in patients with chronic obstructive pulmonary disease. Thorax 1999 Jul;54(7):581-586 [FREE Full text] [CrossRef] [Medline]
    14. Paternostro-Sluga T, Grim-Stieger M, Posch M, Schuhfried O, Vacariu G, Mittermaier C, et al. Reliability and validity of the Medical Research Council (MRC) scale and a modified scale for testing muscle strength in patients with radial palsy. J Rehabil Med 2008 Aug;40(8):665-671 [FREE Full text] [CrossRef] [Medline]
    15. Chawla NV, Bowyer KW, Hall LO, Kegelmeyer WP. SMOTE: Synthetic Minority Over-sampling Technique. jair 2002 Jun 01;16:321-357. [CrossRef]
    16. Verbiest N, Ramentol E, Cornelis C, Herrera F. Preprocessing noisy imbalanced datasets using SMOTE enhanced with fuzzy rough prototype selection. Applied Soft Computing 2014 Sep;22:511-517. [CrossRef]
    17. Yijing L, Haixiang G, Xiao L, Yanan L, Jinling L. Adapted ensemble classification algorithm based on multiple classifier system and feature selection for classifying multi-class imbalanced data. Knowledge-Based Systems 2016 Feb;94:88-104. [CrossRef]
    18. Douzas G, Bacao F, Last F. Improving imbalanced learning through a heuristic oversampling method based on k-means and SMOTE. Information Sciences 2018 Oct;465:1-20. [CrossRef]
    19. Burman P. A Comparative Study of Ordinary Cross-Validation, v-Fold Cross-Validation and the Repeated Learning-Testing Methods. Biometrika 1989 Sep;76(3):503. [CrossRef]
    20. Snoek J, Larochelle H, Adams R. Practical Bayesian optimization of machine learning algorithms. 2012 Presented at: Annual Advances in Neural Information Processing Systems 2012; Dec 3-6, 2012; Lake Tahoe, NV   URL:
    21. MATLAB R2020a.   URL: [accessed 2020-08-25]
    22. Hacke W, Kaste M, Bluhmki E, Brozman M, Dávalos A, Guidetti D, et al. Thrombolysis with Alteplase 3 to 4.5 Hours after Acute Ischemic Stroke. N Engl J Med 2008 Sep 25;359(13):1317-1329. [CrossRef]
    23. Goyal M, Menon BK, van Zwam WH, Dippel DWJ, Mitchell PJ, Demchuk AM, et al. Endovascular thrombectomy after large-vessel ischaemic stroke: a meta-analysis of individual patient data from five randomised trials. The Lancet 2016 Apr;387(10029):1723-1731. [CrossRef]
    24. Goyal M, Jadhav AP, Wilson AT, Nogueira RG, Menon BK. Shifting bottlenecks in acute stroke treatment. J Neurointerv Surg 2016 Dec;8(11):1099-1100. [CrossRef] [Medline]
    25. Berglund A, Svensson L, Sjöstrand C, von Arbin M, von Euler M, Wahlgren N, et al. Higher Prehospital Priority Level of Stroke Improves Thrombolysis Frequency and Time to Stroke Unit. Stroke 2012 Oct;43(10):2666-2670. [CrossRef]
    26. Berglund A, von Euler M, Schenck-Gustafsson K, Castrén M, Bohm K. Identification of stroke during the emergency call: a descriptive study of callers' presentation of stroke. BMJ Open 2015 May 28;5(4):e007661 [FREE Full text] [CrossRef] [Medline]
    27. Sukumaran M, Cantrell D, Ansari S, Huryley M, Shaibani A, Potts M. Stroke patient workflow optimization. Endovasc Today 2019 Feb;18(2):46-50 [FREE Full text]
    28. Itrat A, Taqui A, Cerejo R, Briggs F, Cho S, Organek N, Cleveland Pre-Hospital Acute Stroke Treatment Group. Telemedicine in Prehospital Stroke Evaluation and Thrombolysis: Taking Stroke Treatment to the Doorstep. JAMA Neurol 2016 Mar;73(2):162-168. [CrossRef] [Medline]
    29. Hess DC, Audebert HJ. The history and future of telestroke. Nat Rev Neurol 2013 Jul;9(6):340-350. [CrossRef] [Medline]
    30. Andsberg G, Esbjörnsson M, Olofsson A, Lindgren A, Norrving B, von Euler M. PreHospital Ambulance Stroke Test - pilot study of a novel stroke test. Scand J Trauma Resusc Emerg Med 2017 May 11;25(1):37 [FREE Full text] [CrossRef] [Medline]
    31. Anderson ER, Smith B, Ido M, Frankel M. Remote assessment of stroke using the iPhone 4. J Stroke Cerebrovasc Dis 2013 May;22(4):340-344. [CrossRef] [Medline]
    32. Alber M, Buganza Tepole A, Cannon WR, De S, Dura-Bernal S, Garikipati K, et al. Integrating machine learning and multiscale modeling-perspectives, challenges, and opportunities in the biological, biomedical, and behavioral sciences. NPJ Digit Med 2019;2:115 [FREE Full text] [CrossRef] [Medline]
    33. Held JPO, Klaassen B, Eenhoorn A, van Beijnum BJF, Buurke JH, Veltink PH, et al. Inertial Sensor Measurements of Upper-Limb Kinematics in Stroke Patients in Clinic and Home Environment. Front Bioeng Biotechnol 2018;6:27 [FREE Full text] [CrossRef] [Medline]
    34. Ertzgaard P, Öhberg F, Gerdle B, Grip H. A new way of assessing arm function in activity using kinematic Exposure Variation Analysis and portable inertial sensors--A validity study. Man Ther 2016 Mar;21:241-249. [CrossRef] [Medline]
    35. Salvi D, Poffley E, Orchard E, Tarassenko L. The Mobile-Based 6-Minute Walk Test: Usability Study and Algorithm Development and Validation. JMIR Mhealth Uhealth 2020 Jan 03;8(1):e13756 [FREE Full text] [CrossRef] [Medline]
    36. Gubbi J, Rao AS, Fang K, Yan B, Palaniswami M. Motor recovery monitoring using acceleration measurements in post acute stroke patients. BioMed Eng OnLine 2013;12(1):33. [CrossRef]
    37. Lei H, Huang Z, Zhang J, Yang Z, Tan E, Zhou F, et al. Joint detection and clinical score prediction in Parkinson's disease via multi-modal sparse learning. Expert Systems with Applications 2017 Sep;80:284-296. [CrossRef]
    38. Campanella G, Hanna MG, Geneslaw L, Miraflor A, Werneck Krauss Silva V, Busam KJ, et al. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nat Med 2019 Aug;25(8):1301-1309 [FREE Full text] [CrossRef] [Medline]
    39. Nemati S, Holder A, Razmi F, Stanley MD, Clifford GD, Buchman TG. An Interpretable Machine Learning Model for Accurate Prediction of Sepsis in the ICU. Critical Care Medicine 2018;46(4):547-553. [CrossRef]
    40. Kim J, Chae M, Chang H, Kim Y, Park E. Predicting Cardiac Arrest and Respiratory Failure Using Feasible Artificial Intelligence with Simple Trajectories of Patient Data. J Clin Med 2019 Aug 29;8(9):1336 [FREE Full text] [CrossRef] [Medline]
    41. Li Q, Li Y, Gao J, Su L, Zhao B, Demirbas M, et al. A confidence-aware approach for truth discovery on long-tail data. Proc VLDB Endow 2014 Dec;8(4):425-436. [CrossRef]
    42. Hulsen T, Jamuar SS, Moody AR, Karnes JH, Varga O, Hedensted S, et al. From Big Data to Precision Medicine. Front Med (Lausanne) 2019;6:34 [FREE Full text] [CrossRef] [Medline]
    43. Jiang F, Jiang Y, Zhi H, Dong Y, Li H, Ma S, et al. Artificial intelligence in healthcare: past, present and future. Stroke Vasc Neurol 2017 Dec;2(4):230-243 [FREE Full text] [CrossRef] [Medline]
    44. Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA 2016 Dec 13;316(22):2402-2410. [CrossRef] [Medline]
    45. Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature 2017 Jan 25;542(7639):115-118. [CrossRef]
    46. Qassem T. Emerging technologies for dementia patient monitoring. In: Xhafa F, Moore P, Tadros G, editors. Advanced Technological Solutions for E-Health and Dementia Patient Monitoring. Dallas, TX: IGI Global; 2015:62-104.
    47. Seiffert C, Khoshgoftaar TM, Van Hulse J, Napolitano A. RUSBoost: A Hybrid Approach to Alleviating Class Imbalance. IEEE Trans Syst, Man, Cybern A 2010 Jan;40(1):185-197. [CrossRef]
    48. Ammar H, Eaton E, Ruvolo P, Taylor M. Unsupervised cross-domain transfer in policy gradient reinforcement learning via manifold alignment. 2015 Presented at: Twenty-Ninth AAAI Conference on Artificial Intelligence; Jan 25-30, 2015; Austin, TX.
    49. Pan SJ, Yang Q. A Survey on Transfer Learning. IEEE Trans Knowl Data Eng 2010 Oct;22(10):1345-1359. [CrossRef]
    50. Mikołajczyk A, Grochowski M. Data augmentation for improving deep learning in image classification problem. 2018 Presented at: 2018 International interdisciplinary PhD workshop (IIPhDW); May 9-12, 2018; Swinoujscie, Poland. [CrossRef]
    51. Huang C, Li Y, Chen CL, Tang X. Deep Imbalanced Learning for Face Recognition and Attribute Prediction. IEEE Trans Pattern Anal Mach Intell 2019:1-1. [CrossRef]
    52. Knoll F, Hammernik K, Kobler E, Pock T, Recht MP, Sodickson DK. Assessment of the generalization of learned image reconstruction and the potential for transfer learning. Magn Reson Med 2019 Jan;81(1):116-128 [FREE Full text] [CrossRef] [Medline]
    53. Huynh BQ, Li H, Giger ML. Digital mammographic tumor classification using transfer learning from deep convolutional neural networks. J Med Imag 2016 Jul 01;3(3):034501. [CrossRef]
    54. Hussain Z, Gimenez F, Yi D, Rubin D. Differential data augmentation techniques for medical imaging classification tasks. 2017 Presented at: AMIA 2017 Annual Symposium Proceedings; Nov 4-8, 2017; Washington, DC.
    55. Shin H, Tenenholtz NA, Rogers JK, Schwarz CG, Senjem ML, Gunter JL, et al. Medical image synthesis for data augmentation and anonymization using generative adversarial networks. 2018 Presented at: International workshop on simulation and synthesis in medical imaging; Sep 6, 2018; Granada, Spain.
    56. Jeni L, Cohn J, De LTF. Facing imbalanced data--recommendations for the use of performance metrics. 2013 Presented at: Humaine association conference on affective computing and intelligent interaction; Sep 2-5, 2013; Geneva, Switzerland. [CrossRef]
    57. López V, Fernández A, García S, Palade V, Herrera F. An insight into classification with imbalanced data: Empirical results and current trends on using data intrinsic characteristics. Information Sciences 2013 Nov;250:113-141. [CrossRef]


    AUC: area under the receiver operating characteristic curve
    EMS: emergency medical service
    EVT: endovascular thrombectomy
    IV tPA: intravenous tissue plasminogen activator
    K-NN: K-nearest neighbor
    MRC: Medical Research Council
    NIHSS: National Institutes of Health Stroke Scale
    SMOTE: synthetic minority oversampling technique
    SVM: support vector machine

    Edited by G Eysenbach; submitted 28.05.20; peer-reviewed by Y Arai, KL Ong; comments to author 18.06.20; revised version received 13.08.20; accepted 13.08.20; published 16.09.20

    ©Eunjeong Park, Kijeong Lee, Taehwa Han, Hyo Suk Nam. Originally published in the Journal of Medical Internet Research (, 16.09.2020.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.