Application of an Adaptive, Digital, Game-Based Approach for Cognitive Assessment in Multiple Sclerosis: Observational Study

Background: Cognitive impairment is one of the most debilitating manifestations of multiple sclerosis. Currently, the assessment of cognition relies on a time-consuming and extensive neuropsychological examination, which is only available in some centers. Objective: To enable simpler, more accessible cognitive screening, we sought to determine the feasibility and potential assessment sensitivity of an unsupervised, adaptive, video game–based digital therapeutic to assess cognition in multiple sclerosis. Methods: A total of 100 people with multiple sclerosis (33 with cognitive impairment and 67 without cognitive impairment) and 24 adults without multiple sclerosis were tested with the tablet game (EVO Monitor) and standard measures, including the Brief International Cognitive Assessment for Multiple Sclerosis (which included the Symbol Digit Modalities Test [SDMT]) and Multiple Sclerosis Functional Composite 4 (which included the Timed 25-Foot Walk test). Patients with multiple sclerosis also underwent neurological evaluations and contributed recent structural magnetic resonance imaging scans. Group differences in EVO Monitor performance and the association between EVO Monitor performance and standard measures were investigated. Results: Participants with multiple sclerosis and cognitive impairment showed worse performance in EVO Monitor compared with participants without multiple sclerosis (P=.01) and participants with multiple sclerosis without cognitive impairment (all P<.002). Regression analyses indicated that participants with a lower SDMT score showed lower performance in EVO Monitor (r=0.52, P<.001). Further exploratory analyses revealed associations between performance in EVO Monitor and walking speed (r=–0.45, P<.001) as well as brain volumetric data (left thalamic volume: r=0.47, P<.001; right thalamic volume: r=0.39, P=.002; left rostral middle frontal volume: r=0.28, P=.03; right rostral middle frontal volume: r=0.27, P=.03). Conclusions: These findings suggest that EVO Monitor, an unsupervised, video game–based digital program integrated with adaptive mechanics, is a clinically valuable approach to measuring cognitive performance in patients with multiple sclerosis. Trial Registration: ClinicalTrials.gov NCT03569618; https://clinicaltrials.gov/ct2/show/NCT03569618 (J Med Internet Res 2021;23(1):e24356) doi: 10.2196/24356


Introduction
Cognitive impairment (CI) occurs in 30% to 70% of people with multiple sclerosis (MS) and has a profound influence on a patient's personal functioning, social interaction, employment, and overall quality of life [1,2]. Being able to effectively detect CI is essential to better managing further decline [3] and helping patients navigate problems related to their daily living. The most commonly affected cognitive domains are processing speed, attention, executive function, and memory [4,5]. Currently, clinical cognitive assessment relies on a comprehensive neuropsychological examination, which is time-consuming and extensive. The examination results may be affected by patient fatigue or loss of engagement. Given the large interindividual variability in the pattern of CI in MS [6,7], the traditional nonadaptive assessments may overlook cognitive deficiencies [8]. Taking advantage of technology, digital tools offer a platform to integrate personalizing features, including adaptive staircase algorithms to video game-style mechanics for cognitive assessment [8][9][10][11][12][13][14][15]. This approach can mitigate potential ceiling and floor effects when interindividual variability is high, and leads to more reliable assessments that can be completed in a timely manner [8,16]. Furthermore, digital tools can be easily applied in different settings, including patients' homes, which substantially improves health care accessibility for patients who have difficulties with travel to the clinic due to cognitive or physical disabilities [14,17,18].
Digital tools have been used for functional assessment, rehabilitation, and health care monitoring in clinical populations such as stroke [19,20], schizophrenia [21], depression [10,12], attention-deficit/hyperactivity disorder [15,[22][23][24], and neurodevelopmental disorders [8,9]. In MS, studies have demonstrated that by using digital tools, health care services can be delivered effectively [25,26] and even that comprehensive neurological exams can be performed remotely [27]. However, to the best of our knowledge, applications of serious video game-based digital tools incorporating closed-loop adaptation mechanics as cognitive assessments for people with MS have not been assessed.
We previously evaluated whether a video game-based digital tool, EVO-AKL-T01 (Akili), could improve cognition in MS [17,18]. After these studies, we looked to better characterize cognitive function using similar tools based on these cognitive therapeutics. EVO Monitor (Akili) was developed based on findings that a precursor (NeuroRacer), embedded with adaptive algorithms, is sensitive to age-related cognitive decline across the lifespan and can enhance cognitive control [28]. EVO Monitor, which includes some of NeuroRacer's early closed-loop features, is a novel, tablet-based, digital platform that is incorporated with video game mechanisms, visual and auditory feedback, adaptive algorithms, and sophisticated graphics that is designed to assess executive function, attention, and information processing speed for clinical populations. Specifically, it consists of 3 tasks: perceptual discrimination, visuomotor tracking, and multitasking. In the perceptual discrimination task, the participants complete a go/no go-like paradigm, in which they tap the iPad screen for correctly colored target stimuli while ignoring distracting targets. The visuomotor tracking task requires the participant to tilt the iPad to steer an avatar around obstacles. The multitasking task requires participants to perform both perceptual discrimination and visuomotor tracking concurrently. The tool is designed to challenge attention, goal management, and information processing speed in the setting of interference. EVO Monitor has been validated as a tool to assess cognitive ability in children with and without neurodevelopmental disabilities. For example, it can differentiate between the performance of children with or without 16p11.2 BP4-BP5 deletion and healthy controls, whereas traditional nonadaptive cognitive assessments overlook group differences [8].
In this study, we aimed to determine the feasibility and potential assessment sensitivity of EVO Monitor as an unsupervised, tablet game-based approach with adaptive algorithms to assess cognition in MS. In this context, 100 people with MS and 24 adults without MS were tested with EVO Monitor and standard measures, including the written version of Symbol Digit Modalities Test (SDMT), the most sensitive test for detecting cognitive involvement in the MS course [29,30]. Group-level differences were examined to evaluate whether EVO Monitor is sensitive to differences among participants with MS with and without CI as well as participants without MS. The association between performance in EVO Monitor and standard measures was investigated. Furthermore, since cognition in MS has been shown to correlate with physical fitness, where aerobic capacity and muscular strength outcomes are associated with cognitive processing speed and inhibitory control [31,32], and magnetic resonance imaging (MRI) measures have shown that thalamus damage is associated with the presence of CI, and frontal lesion is associated with executive function [33,34], exploratory analyses of the correlation between performance in EVO Monitor and both physical measures and MRI volumetric data (thalamus and frontal lobes) were conducted to understand whether cognitive performance assessed by a video game-based digital tool (EVO Monitor) would be associated with physical activity and structural MRI measures.

Participants
A total of 100 adults with a diagnosis of clinically isolated syndrome or MS [35] (mean age: 52.2, standard error of the mean [SEM] 1.24 years) were recruited from the University of California, San Francisco (UCSF) Multiple Sclerosis and Neuroinflammation Center. Patients with clinical relapses or steroid use in the past month or with severe visual, cognitive, or motor impairment that would preclude the use of a tablet-based tool were excluded. A group of 24 adults without MS (non-MS) (mean age: 46.0, SEM 3.72 years) with no chronic autoimmune diseases were also recruited from the UCSF staff, willing family members of patients in the clinic, and other eligible and willing volunteers. All participants with MS were recruited as part of studies to determine the feasibility [18] and preliminary efficacy [17] of the EVO platform as a digital therapeutic to improve processing speed in people with MS. The analysis of this study was based on baseline performance data (ie, before any cognitive intervention) of our feasibility [18] and efficacy [17] trials. All procedures performed in the study involving human participants were approved by the Committee for Human Research at the University of California, San Francisco. Written informed consent was obtained from each participant. The trial is registered with ClinicalTrials.gov (NCT03569618).

Standard Measures
The Brief International Cognitive Assessment for Multiple Sclerosis is a cognitive assessment tool that is validated in MS populations as compared with participants without MS [36]. It is a standardized, internationally validated battery [37] including (1) the SDMT, a widely used measure of attention and information processing speed in MS [29,30]; (2) the California Verbal Learning Test Second Edition, a verbal memory immediate recall test [38]; and (3) the Brief Visuospatial Memory Test Revised, a visual memory immediate recall test [39]. Serial versions of all tests were used to minimize practice effects [40].

EVO Monitor
EVO Monitor is a digital cognitive assessment developed by Akili Interactive Labs (Akili, Boston) to assess cognitive function, including attention and related cognitive control processes in clinical populations ( Figure 1). The program is an immersive action video game that has been engineered with adaptive algorithms to target fronto-parietal brain networks fundamentally linked to attentional control among other aspects of cognition. It was developed based on the principles of NeuroRacer, an innovative cognitive intervention that is sensitive to age-related cognitive decline [28]. EVO Monitor comprises 3 tasks: perceptual discrimination, visuomotor tracking, and multitasking. In the perceptual discrimination task, the participants are instructed to respond to colored target stimuli by tapping the iPad screen while ignoring distractors. In the visuomotor tracking task, the participants navigate a character along a dynamically moving road while avoiding walls and obstacles by tilting the iPad. The multitasking task requires participants to perform both perceptual discrimination and visuomotor tracking at the same time until participants complete a minimum number of trials and reach a stable level of performance. The adaptive algorithms change game difficulty on a trial-by-trial basis for both the perceptual discrimination (adapting the response window for a target) and visuomotor tracking (adapting the speed of the forward path), with real-time feedback making the participants aware of their performance. More specifically, the closed-loop adaptive algorithm makes proportional changes in gameplay difficulty to keep the player's performance at an approximate 80% rate of accuracy based on adaptive psychometric principles [43][44][45], which ensures that task difficulty is equated across participants [45,46] and enhances participant engagement [47]. It takes approximately 7 minutes to complete the tasks, with a specified number of both correct and incorrect trials allowing the adaptive algorithm to settle on a prescribed level of difficulty (ie, threshold level) that would converge on a consistent accuracy rate. The threshold level represents task performance, as it indicates the task difficulty at which the participants achieve approximately an 80% rate of accuracy.
At the beginning of the EVO Monitor session, a brief instruction is given on the iPad screen with a short practice to ensure the participant understands the tasks, followed by one session of the actual assessment, which takes about 7 minutes, with a task order of multitasking, perceptual discrimination, visuomotor tracking, and multitasking. Each task is about 1.5 to 2 minutes with no break in between. Although the assessment is self-guided, a study coordinator sat in with participants to ensure they were following the instructions. None of the participants had played EVO Monitor before participation. Since EVO Monitor continuously monitors the user's performance at a rate of 30 frames per second, the measured reaction time, perceptual discrimination task sensitivity (eg, hit and false alarm rates), and visuomotor tracking performance generate 39 basic performance metrics. Among the 39 metrics extracted from participants' navigation of EVO Monitor, we prespecified the calculated threshold levels during both single (perceptual discrimination, visuomotor tracking) and multitasking conditions as attentional measures, according to previous studies [8,16].

Basic Reaction Time
To ensure that any observed differences in cognitive measures between groups were not due to differences in motoric speed, we assessed basic response speed of participants on a simple task with minimal loading on executive function skills [28]. This task was designed to index the motoric speed, and the measured data were only included in the analyses as a covariate to control for potential motor speed deficits in participants with MS. In this task, participants were instructed to respond to a target stimulus (40 trials) as fast as they could by tapping a button on an iPad platform. Similar to EVO Monitor, this task used adaptive algorithms that modulate the challenge level of the task on a trial-by-trial basis based on individual performance. Only data from the dominant hand were included as each participant's basic reaction time (BRT) in the following analyses.

Clinically Acquired MRI Measures
Clinically or research-acquired brain isotropic T1 and T2 fluid-attenuated inversion recovery images were available for 56 participants with MS (16/56, 29% with 1.5-T and 40/56, 71% with 3-T MRIs) at a mean of 76.0 (SEM 33.9) days before the study visit. Lesion segmentation was performed using the LST (lesion segmentation toolbox) lesion probability algorithm 2.0 DICOM (Digital Imaging and Communications in Medicine) v1.4 segmentation pipeline, which creates lesion probability maps, masks, and labels. These were then manually validated by an expert radiologist (SS). Volumetric analysis was performed from T1 anatomical images using 3 complementary tools: FreeSurfer 5.3 and ANTs Morphology 2.1.0 [48], used to segment tissue into cerebrospinal fluid, cortical grey matter, subcortical grey matter, white matter, brainstem, and cerebellum, and Mindboggle 1.0 [49], which combines the morphology outputs of FreeSurfer and ANTs to generate volume images and tabular information for further analysis. Bilateral thalamic and frontal lobe volumetric measures were normalized to individual intracranial volume [50] for exploratory analyses of association between performance in EVO Monitor and MRI volumetric measures. The selection of the thalamus and frontal lobe as regions of interest was based on ample evidence indicating an association between atrophy in these regions and cognitive dysfunction in MS [33,34].

Statistical Analysis
In order to evaluate whether EVO Monitor is sensitive to differences among participants with MS with and without CI as well as participants without MS, participants with MS were divided into 2 subgroups (ie, CI and non-CI) according to their baseline SDMT z scores. We characterized the participants with a SDMT z score lower than -1 based on published normative data [51] as having CI. Group-level differences were assessed with one-way analysis of covariance (ANCOVA) with age, sex, years of education, and BRT as covariates to control for potential differences in demographic features and motoric quickness. Two-tailed Student t tests were carried out for post hoc comparisons when appropriate. To discern the association between EVO Monitor and standard measures, Pearson correlation analyses were performed between performance in EVO Monitor and SDMT. In an exploratory analysis, the correlation between performance in EVO Monitor and both physical MS measures (T25FW) and MRI volumetric data (thalamus and frontal lobes) was assessed with Pearson correlation analyses. Partial correlation analyses including age, sex, years of education, and BRT as covariates were applied when appropriate. All numerical data are presented as the mean and SEM. The statistical analyses were performed using IBM SPSS Statistics version 22.0 (IBM Corp). The significance of the statistical level was set at P≤.05.

Participant Characteristics
A total of 100 participants with MS and 24 participants without MS were enrolled in the study. For analysis purposes, the 100 participants with MS were divided into CI (n=33) and non-CI (n=67) subgroups. Table 1 summarizes their clinical and demographic characteristics. Figure 2 details the completion rate for each test.

Group Differences
To evaluate whether EVO Monitor is sensitive to differences between participants with MS with and without CI as well as participants without MS, one-way ANCOVA with age, sex, years of education, and BRT as covariates was performed for the threshold level, which reflects task performance. Significant group differences in multitasking (F 2,109 =8. 33 (Figure 3). These findings indicate that EVO Monitor, a video game-based assessment designed to assess executive function, attention, and information processing speed, is sensitive to capture group-level differences between participants with MS with or without CI as well as participants without MS.

Association Between Performance in EVO Monitor and Standard Measures
Pearson correlation analyses were performed to scrutinize associations between performance in EVO Monitor and standard cognitive measures. The SDMT showed significant correlations with the EVO multitasking threshold level ( Figure 4 and Table  2). Including age, sex, years of education, and BRT as covariates did not change the results (Table 2). Restricting the analyses to only participants with MS showed similar results. Furthermore, associations between clinical characteristics (ie, Expanded Disability Status Scale [EDSS] and disease duration) and the EVO multitasking threshold level were observed. Analyses of EVO perceptual discrimination and visuomotor tracking threshold levels showed similar results ( Table 2).  In a planned exploratory analysis, we investigated the correlation between performance in EVO Monitor and both MS-related physical function (walking speed, T25FW) and MRI volumetric data, given that these factors have been reported to associate with cognition in MS [31][32][33][34]. A negative correlation was observed between T25FW and multitasking threshold level (r=-0.45, P<.001) ( Figure 5). The results remained similar when including demographic features as covariates (Table 3). Clinically acquired MRI volumetric data were available in 56 participants with MS (17 with CI). We focused on the thalamus and frontal lobes, given their reported relationships with cognition in MS [33,34]. The analyses revealed a positive correlation between the EVO perceptual discrimination threshold level and bilateral thalamic (left: r=0. 47

Discussion
In this study, we aimed to determine whether in-game assessment features of EVO Monitor, an unsupervised, digital, video game-based tool integrated with adaptive algorithms, could represent a novel and sensitive way to perform unsupervised cognitive evaluations in MS. We found a significant group-level difference in performance on EVO Monitor among participants with MS with and without CI as well as participants without MS. Furthermore, we discovered an association between performance in EVO Monitor and standard cognitive measures, physical measures, and MRI volumetric data. These results provide evidence that EVO Monitor, an unsupervised, tablet game-based digital tool designed to assess executive function, attention, and information processing, could effectively assess cognitive performance in people with MS.
We demonstrated that EVO Monitor is sensitive enough to reveal group-level differences, as shown by a lower threshold level in participants with MS with CI compared to both participants with MS without CI and participants without MS in all 3 task conditions. The EVO Monitor program was developed to challenge nearly all aspects of cognition, including executive function, attention, and information processing. In perceptual discrimination and visuomotor tracking tasks, fast information processing is highly demanded, given that the participants are asked to make a fast response to certain stimuli (ie, colored target) and rapidly changing environment (ie, dynamically moving road with walls and obstacles), respectively. To successfully perform the multitasking challenge, there are additional cognitive requirements for selective attention, sustained attention, task switching, and goal management. Since processing speed, attention control, and executive function are the most commonly affected cognitive domains in MS [4,5], the lower EVO Monitor performance in participants with MS and CI may reflect a lower performance in these cognitive domains in general. These results suggest that cognitive dysfunction in MS can be captured by EVO Monitor, a digital tool with adaptive algorithms developed for cognitive assessment.
Using unsupervised digital tools provides a less stressful context and a high level of standardization of assessment. Moreover, it can be easily applied in multiple settings, including patients' homes, which substantially improves access to these therapies for patients who may face scheduling, geographic, or economic barriers in accessing standard forms of cognitive assessment. With the advances in digital therapeutics, assessment and health care services for people with MS have been transitioning to digital platforms [25,26,52]. These results extended the application of digital tools for cognitive assessments in MS by incorporating advanced visualization and reward loops with closed-loop adaptation mechanics to enhance engagement and reduce interindividual variability, which leads to more reliable assessments [8,47].
An association between performance in EVO Monitor and standard measures was observed. Participants with a better performance in EVO Monitor showed higher SDMT scores. SDMT is considered the most sensitive measurement for the evaluation of cognitive involvement and information processing speed in the MS course [29,30]. While the participant substitutes geometric symbols for numbers while scanning a response key, a number of domains-processing speed, sustained attention, visual scanning and tracking, and motoric quickness-are being challenged. Although the task structures are different between EVO Monitor and SDMT, the shared cognitive functions subserving the two tasks may explain the correlational links. The results suggest that EVO Monitor can reliably reflect performance in cognitive tasks that involve information processing speed, selective attention, sustained attention, visual tracking, and goal management. Importantly, analyses including only participants with MS did not change the results but further revealed a negative correlation between clinical characteristics (ie, EDSS and disease duration) and performance in EVO Monitor, where participants with MS who had a longer disease duration and higher level of disability showed a lower performance in EVO Monitor. Although the correlation was modest, given that cognitive deficits tend to progress with disease duration [53] and disability progression [54] in MS, the observed association was expected.
In an exploratory analysis, we discovered an association between EVO Monitor performance and the T25FW as well as the bilateral rostral middle frontal and thalamic volume. In line with previous studies reporting that cognition in MS is associated with physical fitness and balance [55,56] and the concept of cognitive-motor coupling [57], in this study we found that better EVO Monitor performance is correlated with a faster walking speed as measured by the T25FW. Imaging [58] and postmortem [59] studies have suggested that frontal and subcortical regions involved in executive function and cognitive processing speed are also related to the spatial and temporal aspects of gait. The observed association between EVO Monitor performance and the T25FW may be explained by cerebral injury causing impairment in both domains in MS.
Additionally, the performance in the perceptual discrimination task of EVO Monitor positively correlated with rostral middle frontal and thalamic volume. The rostral middle frontal region is part of the dorsolateral prefrontal cortex, an area that is considered the center of executive function [60] and several domains of cognitive control abilities required to perform the EVO perceptual discrimination task, including sustained attention [61], selective attention [62], and inhibitory control [63]. Studies have also demonstrated that the thalamus mediates arousal states [64], and thalamic atrophy is the most significant MRI correlate of CI in MS [65,66]. It is unclear why performance in multitasking, a task including both perceptual discrimination and visuomotor tracking, did not correlate with thalamus and frontal volumetric data. Given that the posterior cortex has been linked to sustaining attention to spatial locations [67], a key component of the visuomotor task, it is possible that the multitasking task does not solely rely on frontal and thalamic resources but is also supported by the posterior cortex. Therefore, the examined correlation was not strong enough to be detected, as it was in the perceptual discrimination task.
Although our analyses were exploratory, the observed correlations support the need to better understand how EVO Monitor performance is associated with physical performance and the structural and functional changes of the brain in people with MS. Specifically, studies with functional MRI and connectivity data would provide essential information about neural correlates of the EVO Monitor tasks and, more importantly, pathological changes related to CI and neural plastic changes as a result of cognitive remediation.
Previously, we showed that EVO-AKL-T01, a video game-based digital tool similar to EVO Monitor, is an effective in-home cognitive remediation program for MS [17,18]. The high adherence rate during the 4-to 6-week home-based cognitive rehabilitation strategy indicated that remote digital tools are well accepted by patients with MS, who may have limited access to cognitive assessment or treatment. Since EVO Monitor is a digital tool designed as a self-guided assessment, it can be used as an in-home cognitive evaluation with multiple assessments to track either the progression of CI or the responsiveness to cognitive interventions, which can substantially help patients navigate problems related to cognitive issues. Future studies are warranted to evaluate the use of EVO Monitor in everyday situations across different contexts and investigate whether the results would be different due to fluctuations in cognitive ability during the day. Another key factor in cognitive assessment is patient fatigue. Well-designed studies that control for impacts of perceived fatigue and fatigability on cognitive assessments measured by digital tools are needed. Future studies comparing how different aspects of fatigue (eg, physical vs cognitive fatigue) affect cognitive performance as assessed by a comprehensive neuropsychological examination and by a digital tool that provides a shorter assessment time would provide additional insight into the role of fatigue in digital cognitive assessments.
There are some limitations to this study. Since we only included one time point in this cross-sectional analysis, it is difficult to determine the reproducibility of the observed results to conclude the test-retest reliability of EVO Monitor. In the exploratory analysis of the association between performance in EVO Monitor and MRI volumetric measures, clinically acquired MRI scans were only available for about half of the participants with MS (n=56), and the impact of acquisition protocol heterogeneity on our MRI metrics should be taken into account while a robust image processing pipeline was applied. The age difference between the non-MS group and non-CI MS group is one of the caveats in this study. However, in our analyses, age was included as a covariate to control for the potential influence of age difference on the results. Skills using digital tools may be a confounding factor influencing the results, as participants with better digital tool skills or more experience using tablet devices may have performed better. Future studies investigating digital tools should control for participants' experience and skills using digital devices.
This study extended the application of digital tools for cognitive assessments in MS by incorporating built-in adaptive staircase algorithms to enhance engagement and mitigate interindividual variability. Furthermore, the encouraging findings suggest that EVO Monitor, an unsupervised, tablet game-based program, is a clinically valuable approach to capturing CI in MS. Since CI is one of the most debilitating manifestations of MS and traveling to clinics may be burdensome due to deficits in mobility or cognition, some patients may have limited access to cognitive assessments. The application of digital cognitive assessments provides flexibility, as the testing can be performed in different settings, including patients' homes. The development of digital cognitive assessments helps patients effectively detect CI and navigate cognition-related problems in their daily living. Future studies with multiple points of data collection and a deeper investigation of how physical performance and the functional and structural changes of the brain affect cognitive performance as measured by digital tools are warranted to provide additional insight.