This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Defining the transition from relapsing-remitting multiple sclerosis (RRMS) to secondary progressive multiple sclerosis (SPMS) can be challenging and delayed. A digital tool (MSProDiscuss) was developed to facilitate physician-patient discussion in evaluating early, subtle signs of multiple sclerosis (MS) disease progression representing this transition.
This study aimed to determine cut-off values and corresponding sensitivity and specificity for predefined scoring algorithms, with or without including Expanded Disability Status Scale (EDSS) scores, to differentiate between RRMS and SPMS patients and to evaluate psychometric properties.
Experienced neurologists completed the tool for patients with confirmed RRMS or SPMS and those suspected to be transitioning to SPMS. In addition to age and EDSS score, each patient’s current disease status (disease activity, symptoms, and its impacts on daily life) was collected while completing the draft tool. Receiver operating characteristic (ROC) curves determined optimal cut-off values (sensitivity and specificity) for the classification of RRMS and SPMS.
Twenty neurologists completed the draft tool for 198 patients. Mean scores for patients with RRMS (n=89), transitioning to SPMS (n=47), and SPMS (n=62) were 38.1 (SD 12.5), 55.2 (SD 11.1), and 69.6 (SD 12.0), respectively (
The MSProDiscuss tool differentiated RRMS patients from SPMS patients with high sensitivity and specificity. In clinical practice, it may be a useful tool to evaluate early, subtle signs of MS disease progression indicating the evolution of RRMS to SPMS. MSProDiscuss will help assess the current level of progression in an individual patient and facilitate a more informed physician-patient discussion.
Multiple sclerosis (MS) is the most common acquired chronic degenerative disease of the central nervous system in young adults, with more than 2.3 million people affected by the disease worldwide [
In previous research, physicians confirmed an unmet need for a tool that could be used in routine clinical practice to raise awareness and facilitate the systematic assessment of early signs of progression to SPMS. Physicians also expressed their preference for a validated digital solution producing an easy to interpret output [
With the preceding in mind, we developed MSProDiscuss, a digital tool to (1) facilitate physician-patient interaction in routine clinical practice; (2) support physicians in evaluating the early signs of progression in a structured, standardized manner based on a patient’s neurological history, the symptoms experienced, and how these affected various domains of the patient’s daily life in the past six months; and (3) help assess patient’s current level of progression. The content of this tool was developed using a mixed methods approach building on quantitative and qualitative assessments. Therefore, for the first time, both patients’ and physicians’ qualitative data were taken into consideration. The summary of the findings from stage 1 (development of the questionnaire) and stage 2 (scoring algorithm) of this comprehensive research is described in
Overview of the development process of draft MSProDiscuss tool (a mixed methods approach).
Twenty physicians (seven from the United States, nine from Germany, and four from Canada) participated in this validation study. Physicians completed a Web-based draft version of the tool for up to 10 patients from their routine practice, comprising three to four patients each with a diagnosis of either RRMS or SPMS, or for patients that they suspected may be progressing to SPMS (“transitioning” patients). Physicians also completed a case report form (CRF) for each patient, which captured physician diagnosis (RRMS, SPMS, or transitioning) and other key clinical information. The order of the CRF and tool were alternated to minimize potential bias in tool completion. Physicians also provided information about their clinical experience by completing a physician CRF and provided their feedback on the content of the draft tool and usefulness of the tool in clinical practice by completing a usability questionnaire.
A target of more than 150 patients was prespecified based on the planned receiver operating characteristic (ROC) analyses. The overall significance level for this study was set at .05, but this value was adjusted using a Bonferroni correction for the sample size calculation to take into account two testing procedures: SPMS versus not SPMS (RRMS and transitioning) and RRMS versus not RRMS (transitioning and SPMS). This led to a sample size that allowed detection of area under the ROC curve (ROC AUC) of at least 0.68 with 90% statistical power with a significance level of .025 [
Specialist neurologists, who were responsible for the care and management of at least five patients with MS per week, were included in this study once they provided written informed consent. The physician was required to be verbally fluent in their local language (either English or German).
As part of the CRF, physicians were required to specify each patient’s clinical diagnosis. RRMS was defined as having a confirmed diagnosis of RRMS according to the 2010 Revised McDonald Criteria [
The draft tool consisted of three sections addressing disease activity, symptoms, and impact of these symptoms on patients’ daily living. Patients’ age, current Expanded Disability Status Scale (EDSS) score [
For validation, two cut-off values were used to visualize the tool output: a score equal or above the upper cut-off indicated SPMS and values lower or equal to the chosen lower cut-off defined RRMS patients. The range between the upper and lower cut-offs indicated transitioning patients (RRMS patients showing early signs of progression but still not classified as SPMS by their treating physician). Following completion of the tool for a patient, an output screen visually displayed the standardized total score linked to a traffic light system, which was also used to obtain physician’s feedback on the usefulness of the tool in clinical practice.
SAS version 9 was used for all statistical analyses.
ROC curve analysis was used to evaluate the sensitivity (true positive rate) and specificity (true negative rate) of different cut-off values on the draft tool. The cut-off values informed the thresholds for which a patient would be classified as RRMS, SPMS, or transitioning. SPMS versus RRMS and transitioning patients were compared to obtain an upper cut-off for the tool, whereas RRMS versus transitioning and SPMS patients were compared to obtain a lower cut-off for the tool. Any value between the lower and upper thresholds was considered indicative of a patient possibly showing signs of progression (ie, in transition to SPMS). The cut-off values were estimated using Youden’s J index [
Two video vignettes depicting scenarios of mock patient-physician interactions were developed (one represented an SPMS patient [
Descriptive data were produced for physician responses to the usability questionnaire. Qualitative responses to the usability questionnaire were coded using thematic analysis methods on ATLAS.ti [
A total of 20 physicians (all neurologists experienced in the treatment of MS) participated in the study. Neurologists reported seeing approximately 19 RRMS patients (range 5-54) and eight SPMS patients (range 1-25) in a week, with an average of 14.8 (SD 11.8) hours per week and an estimated average of 36.6% (SD 27.9%) of their monthly workload dedicated to MS patients (
In total, the neurologists completed the draft tool for 198 MS patients: 89 RRMS, 47 suspected to be transitioning, and 62 SPMS (according to their clinical diagnosis). Patients had a mean age of 44.8 (SD 12.8) years and a mean EDSS score of 4.0 (SD 1.7). The mean duration since RRMS diagnosis was 11.8 (SD 9.2) years—range 7.3 (SD 6.3) years (RRMS) to 17.3 (SD 10.1) years (SPMS)—and mean duration since SPMS diagnosis of 6.3 (SD 5.3) years (range <1-22 years;
Patient demographics and clinical characteristics by physician diagnosis (N=198).a
Patient characteristic | Physician diagnosis | |||
|
RRMS (n=89) | Transitioning (n=47) | SPMS (n=62) | |
|
|
|
|
|
|
n | 89 | 47 | 62 |
|
Mean (SD) | 38.1 (11.3) | 46.2 (10.7) | 53.4 (10.7) |
|
Median (range) | 37 (19-66) | 47 (28-68) | 52 (34-78) |
|
|
|
|
|
|
n | 81 | 47 | 61 |
|
Mean (SD) | 2.6 (1.0) | 4.3 (1.1) | 5.6 (1.4) |
|
Median (range) | 2 (0-7)b,c | 4 (2-7) | 6 (3-9) |
|
||||
|
Yes | 30 | 10 | 9 |
|
No | 59 | 37 | 53 |
|
|
|
|
|
|
n | 89 | 47 | 62 |
|
Mean (SD) | 7.3 (6.3) | 13.2 (8.4) | 17.3 (10.1) |
|
Median (range)d | 5.0 (0-26) | 11.3 (0-37) | 15.8 (0-51) |
|
|
|
|
|
|
n | — | — | 62 |
|
Mean (SD) | — | — | 6.3 (5.3) |
|
Median (range)d | — | — | 5 (0-22) |
aPhysician diagnosis was collected from the patient case report form. EDSS: Expanded Disability Status Scale; RRMS: relapsing-remitting multiple sclerosis; SPMS: secondary progressive multiple sclerosis.
bEDSS=0 (n=1); EDSS=1 (n=0); EDSS=1.5 (n=1).
cLiterature suggests that there are circumstances in which onset of progression to SPMS is early and identifiable at a DSS score of 2 or less [
dMinimum value of 0 indicates a duration of less than 12 months.
According to the data entered into the draft tool (
The impact of symptoms was experienced in all domains of patients’ daily life (
Number of patients experiencing each sign or symptom by physician diagnosis. RRMS: relapsing-remitting multiple sclerosis; SPMS: secondary progressive multiple sclerosis.
Number of patients experiencing each impact by physician diagnosis. RRMS: relapsing-remitting multiple sclerosis; SPMS: secondary progressive multiple sclerosis.
Patients with a physician diagnosis of SPMS scored higher (mean 69.6, SD 12.0) than those patients suspected to be in transition to SPMS (mean 55.2, SD 11.1) and those with a physician diagnosis of RRMS (mean 38.1, SD 12.5,
Distribution of total scores according to physician MS diagnosis. The values within the figure are mean scores.
Youden’s J index and the sum of squares method estimated the optimal cut-off values, with equal weight for sensitivity and specificity (
Cut-off points identified using the Youden’s J index and sum of squares methods.a
Method | Cut-off | AUC (95% CI) | Sensitivity | Specificity | ||
|
|
|
|
|
||
|
|
|
|
|
|
|
|
|
Youden’s J index, sum of squares | 58.9 | 0.91 (0.86–0.95) | 0.82 | 0.84 |
|
|
|
0.86 (0.81–0.91) |
|
|
|
|
|
Sum of squares | 57.8 |
|
0.79 | 0.74 |
|
|
Youden’s J index | 49.5 |
|
0.90 | 0.63 |
|
|
|
|
|
||
|
|
|
0.91 (0.87–0.95) |
|
|
|
|
|
Sum of squares | 51.6 |
|
0.83 | 0.82 |
|
|
Youden’s J index | 53.7 |
|
0.89 | 0.76 |
|
|
|
|
|
|
|
|
|
Youden’s J index, sum of squares | 46.3 | 0.88 (0.84–0.93) | 0.76 | 0.86 |
aAUC: area under the curve; EDSS: Expanded Disability Status Scale; RRMS: relapsing-remitting multiple sclerosis; SPMS: secondary progressive multiple sclerosis.
Receiver operator characteristic (ROC) curves and cut-off points identified by Youden’s J index and sum of squares. AUC: area under the curve; EDSS: Expanded Disability Status Scale; RRMS: relapsing-remitting multiple sclerosis; SPMS: secondary progressive multiple sclerosis.
Sensitivity and specificity by applying the same cut-off points for the draft algorithm with and without EDSS.a
Draft algorithm | Cut-off | Sensitivity | Specificity | |
|
|
|||
|
With EDSS | >57.8 | 0.82 | 0.81 |
|
Without EDSS |
|
0.79 | 0.74 |
|
||||
|
With EDSS | <46.3 | 0.72 | 0.89 |
|
Without EDSS |
|
0.76 | 0.86 |
aEDSS: Expanded Disability Status Scale; RRMS: relapsing-remitting multiple sclerosis; SPMS: secondary progressive multiple sclerosis.
The total score for the draft tool demonstrated excellent interrater reliability (ICC 0.950, 95% CI 0.772-1.000). The ICC was good for the disease activity section (ICC 0.852, 95% CI 0.504-1.000) and the symptoms section (ICC 0.869, 95% CI 0.541-1.000), and close to fair (ICC 0.391, 95% CI 0.073-0.999) for the impacts section.
A statistically significant difference was observed in the total score between EDSS groups and physician diagnosis groups (
The majority of items included in the draft tool showed strong (
The mean time to complete the draft tool was 2.16 minutes per patient (n=83; median 1.59, range 0.48-6.58). Findings from the usability testing are provided in
Findings from the usability testing of the draft tool (N=20).
Usability statements | n (%) | |
Items relevant to identifying progression to SPMSa | 20 (100) | |
Typically collect tool data in clinical practice | 18 (90) | |
|
|
|
|
Anxiety and depression | 2 (10) |
|
New symptoms | 2 (10) |
Time to complete is satisfactory | 17 (85) | |
Traffic light style output is useful and clear | 16 (80) | |
Feasible to implement tool in clinical practice | 17 (85) | |
Open to using the tool as an additional independent evaluation to complement neurological assessment | 17 (85) |
aSPMS: secondary progressive multiple sclerosis.
The MSProDiscuss tool was able to differentiate between RRMS and SPMS patients, with the highest scores seen in patients with a diagnosis of SPMS. The sensitivity for SPMS was consistently around 80% (true positive rate) and specificity (true negative rate) for RRMS above 86%. Overall, the draft tool demonstrated excellent interrater reliability, and good evidence of construct validity using the known-groups method. The neurologists supported the implementation and usefulness of the tool for clinical practice. The items in the draft tool were considered relevant and are typically collected in this setting; hence, they do not represent an additional burden to the clinical practice.
Disease evolution is highly variable in MS, and progression to SPMS is a key milestone for patients. The use of tools supporting the real-time evaluation of early signs of MS progression for use in daily practice is currently an unmet need. A number of studies have investigated predictors of SPMS; however, only a few reported on tools to predict SPMS progression. These tools have been derived using only quantitative empirical assessments of different registry-based databases [
MSProDiscuss was developed using a rigorous mixed methods approach, which incorporated a regression analysis of data from a large observational study and qualitative interviews with patients and physicians. Items included in the tool were previously identified as relevant and suggestive of progressive disease ([
MSProDiscuss is an easy to use physician-completed digital tool intended to facilitate physician-patient dialog in assessing the subtle signs suggestive of disease progression by systematically evaluating and recalling relevant information from patient clinical history, symptoms, and impacts on daily activities experienced over the past six months. Such variables are often assessed in routine clinical practice, but might not be systematically recorded.
In this study, we validated the draft tool and algorithm developed based on the findings from previous studies ([
The impact and severity of symptoms experienced by transitioning and SPMS patients were clearly different compared with patients with RRMS. All symptoms assessed were more frequently reported in SPMS and transitioning patients compared with RRMS patients. Specifically, symptoms related to ambulatory, motor, coordination and balance, bladder and bowel, and cognition were approximately 2.5 to 3.5 times more frequent in SPMS and transitioning patients compared with RRMS patients. These results are consistent with the findings from the qualitative assessment in our previous pilot study ([
The ROC analysis confirmed that the draft tool was able to discriminate between SPMS and RRMS patients with high sensitivity and specificity. Although a stronger performance was observed when EDSS was included in the draft algorithm (AUC 0.905-0.908), the tool also maintained good performance in the absence of EDSS (AUC 0.863-0.882). Cut-offs excluding EDSS were considered appropriate for the final validated tool, as it is intended for use in clinical practice where EDSS might not be routinely assessed. Sensitivity for SPMS was consistently around 80% (true positive rate) and specificity (true negative rate) for RRMS above 86%. The excellent interrater reliability and good evidence of construct validity suggest that the items included in this tool are of relevance to assess early signs of progression. The average time to complete the tool was approximately 2 minutes, and in the usability testing, neurologists supported the implementation and usefulness of the tool for clinical practice.
Some of the analyses were based on physician diagnosis; however, they also completed the tool, which may have introduced some reporting bias. To overcome this potential bias, the order of completion of the patient CRF, including the physician diagnosis and the draft tool, were alternated. Also, neurologists in this study were all well-experienced in diagnosing and managing MS patients; hence, they did not rely on the tool for their diagnosis of SPMS. Importantly, the tool was validated not only against SPMS but also RRMS and patients in transition. Moreover, patients and physicians from different countries were involved (including the United States, Canada, and Germany) to reduce potential bias due to differences in health care systems and approaches adopted to diagnose SPMS, which became evident from previous research ([
The MSProDiscuss tool is now included in a large real-world observational study in Germany (PANGAEA 2.0 [
The aim of MSProDiscuss is to facilitate physician-patient conversation by allowing a comprehensive, but simple, standardized assessment of the patient’s current disease status and level of progression. In this validation study, the MSProDiscuss tool demonstrated its ability to differentiate between patients with RRMS and SPMS with high sensitivity and specificity, with or without EDSS. Thus, the tool will help sensitize for early, subtle signs suggestive of MS disease progression in daily practice. The tool also supports the evaluation of transitioning patients who have not yet converted to SPMS and who might benefit most from optimized early interventions that slow disability accumulation. Evidence from this study suggests the tool may be useful in clinical practice for a more informed physician-patient discussion supporting the successful management of MS.
The final validated MSProDiscuss tool can be accessed on the Neuro-Compass website [
MSProDiscuss: Screenshots of the tool.
Statistical analysis.
Video vignette-SPMS patient.
Video vignette-RRMS patient.
Psychometric properties.
Physician characteristics.
Number of patients experiencing each impact, by severity.
ROC curves for selected cut-off points (applying same cut-offs to algorithm including and excluding EDSS.
Known-groups comparisons for total score.
Known-groups comparisons for section scores.
Item correlations with total score and item correlations with physician diagnosis.
area under the curve
case report form
Expanded Disability Status Scale
interclass correlation coefficient
multiple sclerosis
receiver operating characteristic
relapsing-remitting multiple sclerosis
secondary progressive multiple sclerosis
Funding support was from Novartis Pharma AG, Basel, Switzerland. Adelphi Values (funded by the research sponsor, Novartis) designed and conducted this research along with data collection and management. Both the sponsor and Adelphi Values were involved in data interpretation and the preparation, review, and approval of the study report as well as the publication. The authors gratefully acknowledge all the participating physicians for their contribution and insights on the draft tool. We also acknowledge Uma Kundu of Novartis Healthcare Pvt Ltd, for medical writing support, which included literature search, drafting of the paper, revising the paper as per author comments, and preparation of submission; and Sivaram Vedantam of Novartis Healthcare Pvt Ltd, for scientific editorial review support.
DPM is an employee of Novartis Pharma AG. BB and CJ were employees of Adelphi Values during the conduct of this study, which was paid by Novartis for conducting this research. KT, AT, and CT are employees of Adelphi Values, which was paid by Novartis for conducting this research. TH, FD, and DT are employees of Novartis Pharma AG. MSF and TZ have none to declare.