This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
In chronic neurological diseases, especially in multiple sclerosis (MS), clinical assessment of motor dysfunction is crucial to monitor the disease in patients. Traditional scales are not sensitive enough to detect slight changes. Video recordings of patient performance are more accurate and increase the reliability of severity ratings. When these recordings are automated, quantitative disability assessments by machine learning algorithms can be created. Creation of these algorithms involves non–health care professionals, which is a challenge for maintaining data privacy. However, autoencoders can address this issue.
The aim of this proof-of-concept study was to test whether coded frame vectors of autoencoders contain relevant information for analyzing videos of the motor performance of patients with MS.
In this study, 20 pre-rated videos of patients performing the finger-to-nose test were recorded. An autoencoder created encoded frame vectors from the original videos and decoded the videos again. The original and decoded videos were shown to 10 neurologists at an academic MS center in Basel, Switzerland. The neurologists tested whether the 200 videos were human-readable after decoding and rated the severity grade of each original and decoded video according to the Neurostatus-Expanded Disability Status Scale definitions of limb ataxia. Furthermore, the neurologists tested whether ratings were equivalent between the original and decoded videos.
In total, 172 of 200 (86.0%) videos were of sufficient quality to be ratable. The intrarater agreement between the original and decoded videos was 0.317 (Cohen weighted kappa). The average difference in the ratings between the original and decoded videos was 0.26, in which the original videos were rated as more severe. The interrater agreement between the original videos was 0.459 and that between the decoded videos was 0.302. The agreement was higher when no deficits or very severe deficits were present.
The vast majority of videos (172/200, 86.0%) decoded by the autoencoder contained clinically relevant information and had fair intrarater agreement with the original videos. Autoencoders are a potential method for enabling the use of patient videos while preserving data privacy, especially when non–health-care professionals are involved.
In chronic neurological diseases, especially multiple sclerosis (MS), clinical assessment of motor dysfunction is crucial to monitor the disease in patients [
This study was a subproject of the ASSESS MS study [
A variational autoencoder was trained on 2230 videos comprising the 9 standardized motor performances included in the ASSESS MS study. The autoencoder was structured so that the frames of each video were encoded into a lower-dimensional space and then decoded into their original form.
Structure of the variational autoencoder.
The key property of interest to us was that when a frame is in its coded form, it is computationally prohibited to decipher it without access to the decoder [
Intrarater agreement between the ratings of the original and the decoded videos was assessed using the Cohen weighted kappa with linear weights (ie, disagreements of 1, 2, and 3 were weighted by factors of 1, 2, and 3, respectively). A Cohen kappa of 0 corresponds to chance agreement; 0-0.2, to slight agreement; 0.21-0.4, fair agreement; 0.41-0.6, to moderate agreement; 0.61-0.8, to substantial agreement; and 0.81-1, to almost perfect agreement [
The characteristics of the study population and the participating neurologists are summarized in
In total, 172/200 (86.0%) videos were of sufficient quality to be ratable. The Cohen weighted kappa indicating intra-rater agreement between the original and decoded videos was 0.317. The average difference in the ratings between the original and decoded videos was 0.26, in which the original videos were rated as more severe. The inter-rater agreements of the original and decoded videos were 0.459 and 0.302, respectively. As depicted in
Characteristics of the patients and neurologists who participated in the study.
Characteristic | Value | ||
|
|||
|
Age (years), mean (95% CI) | 44.4 (27-74) | |
|
Gender (female/male), n (%) | 12 (63%)/7 (37%) | |
|
Disease duration (years), mean (95% CI) | 13.2 (1-40) | |
|
Median EDSSa (range) | 3.5 (0-6.5) | |
|
Type of MSb (RRMSc/SPMSd), n (%) | 19 (95%)/1 (5%) | |
|
|||
|
Gender (female/male), n (%) | 5 (50%)/5 (50%) | |
|
Years of experience in neurology, mean (range) | 8.8 (3 to >30) |
aEDSS: Expanded Disability Status Scale.
bMS: multiple sclerosis.
cRRMS: relapsing remitting multiple sclerosis.
dSPMS: secondary progressive multiple sclerosis.
Ratings by 10 neurologists of the original and decoded videos. The colored squares represent the different grades for limb ataxia of the finger-to-nose-test according to the Neurostatus-Expanded Disability Status Scale subscores: black=0, dark grey=1, grey=2, bright grey=3, and white=4. The blue squares represent videos that were judged as not ratable by the neurologists.
In this proof-of-concept study, 172/200 (86.0%) of the decoded videos were of sufficient quality to be ratable. We found fair intrarater agreement between the original and decoded videos. The agreement was better for minor and severe deficits in motor function.
Data security and privacy are increasingly requested by health care professionals for data capture, analysis, and storage [
Patients with MS may present with slight changes in motor performances over their disease course. Clinical assessment of these changes is notoriously difficult. Video analysis of motor performances allows automated analyses and quantification of disability by using machine learning algorithm–based analysis systems such as those used in the ASSESS MS study; however, it requires a huge data set [
A limitation of this proof-of-concept study is the class imbalance of the patient videos according to the four grades of limb ataxia for the finger-to-nose test [
In this proof-of-concept study, we have shown that the vast majority (172/200, 86.0%) of videos decoded by an autoencoder contained clinically relevant information regarding upper extremity motor performance represented by the finger-to-nose test and had fair intrarater agreement. Autoencoders are a potential method for enabling the use of patient videos while preserving data privacy, especially when non–health care professionals are involved.
Expanded Disability Status Scale
multiple sclerosis
relapsing remitting multiple sclerosis
secondary progressive multiple sclerosis
This study was supported by Novartis.
MD received travel support from Bayer AG, Biogen, Teva Pharmaceuticals, and Sanofi Genzyme and research support from University Hospital Basel. CM received travel support from Novartis Pharma AG, Sanofi Genzyme, Teva Pharmaceuticals, and Merck Serono; honoraria for lecturing and consulting from Novartis Pharma AG, Biogen-Idec, and Merck Serono; and compensation for serving on a scientific advisory board from Biogen-Idec, Roche, Merck Serono, and Sanofi Genzyme. JD is an employee of Novartis Pharma AG. AD is an employee of Novartis Pharma AG. CK has received honoraria for lectures and research support from Biogen-Idec, Novartis Pharma AG, Almirall, Bayer Schweiz AG, Teva Pharmaceuticals, Eli Lilly, Merck Serono, Sanofi Genzyme, and the Swiss Multiple Sclerosis Society. SS has received travel support from Bayer, Merck, and Novartis and has received honoraria for consulting from Bayer, Merck, Roche, and Teva. FD is an employee of Novartis Pharma AG. BU has received consultation fees from Biogen-Idec, Novartis Pharma AG, EMD Serono, Teva Pharmaceuticals, Sanofi Genzyme, and Roche. The Multiple Sclerosis Center Amsterdam has received financial support for research from Biogen-Idec, Merck Serono, Novartis Pharma AG, and Teva Pharmaceuticals. In the last 3 years, LK’s institution (University Hospital Basel) received consultancy, steering committee, and advisory board fees from Actelion, Alkermes, Almirall, Bayer, Biogen, Celgene, df-mp, EXCEMED, GeNeuro SA, Genzyme, Merck, Minoryx, Mitsubishi Pharma, Novartis, Roche, Sanofi-Aventis, Santhera, Teva, and Vianex and as well as royalties for Neurostatus products. These fees were used exclusively for research support in the Department of Neurology. For educational activities of the Department, the institution received honoraria from Allergan, Almirall, Bayer, Biogen, EXCEMED, Genzyme, Merck, Novartis, Pfizer, Sanofi-Aventis, Teva, and UCB. MJ is an employee of Microsoft Research.