For decades, neuropsychological methods have been leveraged to understand and characterize the relative strengths and weaknesses of individuals experiencing an array of neuropsychiatric syndromes [1-5]. These profiles have also been shown to predict future deterioration in Alzheimer disease and other cognitive and functional outcomes in schizophrenia, bipolar disorder, depression, and traumatic brain injury [6-8]. However, traditional paper and pencil assessments are lengthy and expensive and require considerable training on the part of the assessor. Given these concerns and the rising availability of personal mobile technologies, a movement to capture reliable cognitive functioning digitally has begun [9-12]. Although remote cognitive technologies are still budding, they promise several benefits in clinical and research settings.

Remote platforms greatly increase the accessibility of psychological testing. The National Institute of Mental Health has established a priority to reach typically underserved populations, such as individuals living with limited physical mobility or in rural areas [13]. Digital measures allow testing to be conducted within the comfort of one’s own home, thereby increasing the ability to serve historically difficult-to-reach patients. Remote testing would also likely reduce the time providers spend scheduling and carrying out in-person assessments. These benefits may also help to reduce assessment costs. Accessibility concerns have taken center stage during the current global pandemic, highlighting the need for reliable, easy-to-use, and remote measures.

In addition to opportunities for accessibility, digital remote methods for cognitive testing encourage the use of longitudinal assessments. Repeated measurements provide a wealth of information above and beyond individual snapshots of performance [14]. A study of individuals at clinical high risk for conversion into a psychotic disorder found that although baseline measures of cognition helped differentiate at-risk individuals from healthy participants, cognitive trajectories followed for 2 years differentiated converters from nonconverters [15]. Another study of healthy older adults found that repeated memory assessments pushed to a handheld device were widely accepted and showed associations with the hippocampal brain structure, whereas typical baseline measures of cognition did not [16].

Although in-person neuropsychological methods unquestionably have their benefits and function optimally in distraction-free testing environments, the generalizability of data collected in these unique settings has been questioned [17]. There is a growing body of literature evaluating assessments that can be administered in people’s daily lives, thus potentially capturing more meaningful data on how they function within their own environments [18,19]. For example, a clinician may be more interested in how their patient performs cognitively while completing their night shift at work rather than in a highly controlled testing center at midday.

Digital methods also provide a means to measure important metrics such as reaction time (RT) and trial-by-trial performance, which have also shown associations with clinical outcomes [20-22]. These potential benefits, alongside the added ability of electronic health record integration, may further facilitate person-centered practice and move us closer toward the goal of precision medicine [23]. However, little to no normative data currently exists for these task batteries, limiting the interpretability of new findings [24].

A particular novel mobile assessment platform implemented by the Posit Science Corporation has shown recent utility capturing cognitive performance differences common to psychiatric disorders, such as schizophrenia and depression, using a neuroscience-informed approach to task development [25,26]. These tasks were designed to capture the variability associated with aberrant neural mechanisms underlying an individual’s ability to flexibly acquire new information and adapt to changing cognitive and emotional demands, integral processes for normative cognitive functioning and general learning ability [26-29]. Sound sweeps, a task of auditory processing speed, showed improvements during a cognitive training intervention in individuals with recent-onset and chronic schizophrenia [30]. The greatest improvement was observed during the first 20 hours of training, followed by a plateau at subsequent assessment points. Intraindividual variability in the time taken to reach this plateau was also associated with the likelihood that the intervention would generalize to other untrained cognitive domains, suggesting a potential target-mediated treatment response. Another study found that improvements in sound sweeps performance after cognitive training were correlated with gains in working memory and global cognition [31]. Despite the apparent utility of these tasks, they have yet to be adequately evaluated in normative data samples.

Therefore, we set out to test a battery of 8 web-based cognitive tasks developed by Posit Science in samples drawn from the general community. Sensory perception, social cognition, and executive functions were digitally assessed in-person with state fair attendees and remotely through testing of college students. In addition, the usability and feasibility of this new battery were investigated in a subset of participants.

All study procedures were approved by the institutional review board of the University of Minnesota (UMN). Recruitment and study participation took place either in-person at the Minnesota State Fair (MSF) or remotely for course credit at UMN. The MSF is the second-most highly attended state fair in the United States, with approximately 2 million Midwesterners visiting every year [32]. State fairgoers represent a wide array of demographic backgrounds [33]. MSF participants were asked to participate if they were aged between 18 years and 80 years (inclusive), and UMN students were asked to participate if they were aged between 18 years and 40 years (inclusive). Inclusion criteria for all participants were (1) no visual, auditory, or motor impairments that would prevent completion of the assessments; (2) fluent and literate in English; and (3) no use of illicit substances or alcohol over the prior 8 hours. We received cognitive data from 816 participants and usability survey responses from 219 participants who participated in a substudy. However, cognitive data from 35 participants who took the battery multiple times were omitted from the primary cognitive analyses below, and survey data were omitted for another 24 participants who completed usability questionnaires multiple times. Therefore, 781 participants were included in the primary analyses, and survey data from 195 participants were inspected (Table 1).

After receiving informed consent, the study participants provided demographic information and completed a battery of cognitive tasks. College students participated remotely via their own personal devices (eg, tablets or personal computers) and were asked to find a stable internet connection to complete the tasks in a quiet, private environment using headphones, preferably over-the-ear headphones. MSF participants completed the study procedures in an enclosed fair structure using lab iPads (Apple Inc) and over-the-ear headphones. Participants were randomly assigned to complete 3 to 4 of the 8 cognitive assessments. State fair participants completed a subset of the measures because of the time limit set by the Driven to Discover State Fair program. A maximum of 30 minutes was allowed for giving consent and the completion of demographic information, cognitive measures, and questionnaires. Participants were randomly assigned to complete 1 of 2 cognitive batteries that were equivalent in length and comparable in terms of the number of auditory, visual, and social cognition tests.

Our work aims to translate measures from basic cognitive neuroscience into short, computerized assessments of discrete cognitive processes that individuals can easily complete with minimal assistance in various settings [25]. These assessments are designed to enable the interpretation of specific deficits that could signal that an individual is experiencing cognitive difficulties and impaired learning ability.

The first step in the development of the assessment suite was to decide on the cognitive domains and processes that are known to play a critical role in an individual’s ability to learn new information, to interact adaptively with cognitive and emotional challenges in the environment, and to adapt to new learning demands. In line with the principles of team science, we integrated theoretical perspectives, technical expertise, and empirical knowledge drawn from a team of cognitive neuroscientists working in human and animal model systems, clinical researchers, and preclinical translational behavioral neuroscientists. Through a consensus-building process, we identified 3 critical neural domains: (1) perceptual processing (sound sweeps and beep seeker), (2) executive functioning (bubble pop, pathfinder, and mind bender), and (3) social-emotional processing (face to face, tap the emotion, and emotional face). Within each domain, we identified constructs, that is, a definable cognitive process that could be measured at the behavioral level and for which there existed clearly hypothesized and measurable neural-circuit mechanisms (eg, for executive functioning, set-shifting). We identified a cognitive neuroscience paradigm that could selectively and parametrically measure each of these constructs at the behavioral level. Guided by item response theory, these assessments use adaptive testing models to adjust the difficulty level according to the user, thereby reducing the duration of the test [34]. Online neurocognitive assessments (ONAs) take an average of 184 (SD 201) seconds to complete. Instructions on sampling the ONAs are presented in Multimedia Appendix 1.

This lab-designed measure asked participants to indicate how much they agreed with a series of 5 statements using a 5-point Likert scale ranging from not at all to very much. The specific questions are shown in Multimedia Appendix 2.

All main outcomes were screened for normality. Although some measures showed skewness and kurtosis (eg, beep seeker, tap the emotion, and face to face), all had adequate variance. The task distributions are shown in Multimedia Appendix 1. Nonparametric statistics were subsequently used for beep seeker, tap the emotion, and face to face. Outliers greater than 2.5 SD from the mean were winsorized and represented less than 1% of the data. For ease of comparison, outputs from sound sweeps, mind bender, and face to face were transformed so that higher scores indicated better performance.

General tendencies, dispersion metrics, and associations with demographic variables were calculated for each ONA. A total of 2 individuals identified as intersex were excluded from the gender comparison of ONA performance because of insufficient power. Given that 6 of 8 ONAs showed significant associations with age, task performance was also summarized across 7 different age ranges. To investigate concurrent and discriminant validity across ONAs, a Spearman rank-order correlation matrix was summarized; however, a factor analysis was not conducted because of inefficient overlap of measures completed across samples and a subsequent lack of power.

In this study, we tested a new web-based neurocognitive assessment battery for individuals from the general community. Normative metrics were collected across 8 tasks measuring auditory discrimination, auditory perception, visual-spatial working memory, visual-spatial learning, cognitive flexibility, emotion detection, emotion identification, and inhibitory control of emotion.

Participants found the assessments to be widely acceptable across a range of ages. Age was negatively associated with the extent to which participants were able to understand task instructions; however, the broad majority of individuals in older age groups found the tasks understandable. Still, future iterations of these tasks should consider adaptations to better reach these individuals, such as improving task instructions, increasing practice trials, embedding video tutorials, or lowering the starting level of difficulty to reduce initial frustration. Despite finding the tasks widely interesting and enjoyable, fewer individuals indicated an interest in playing games at home. This difference may suggest that fewer participants would choose to engage with these tasks as a leisure activity but may be open to completing them if recommended by a member of their health care team. It is important to note that contrary to the survey wording, these tasks are not considered to be games and should not be interpreted as such. In contrast to traditional neuropsychological assessments, these measures are brief, allowing clinicians to collect a wealth of cognitive information in a relatively short period. These findings highlight the acceptability and efficiency of the battery.

Participant age was associated with 6 of the 8 tasks, with younger participants performing better than older participants. This trend has been widely observed in the cognitive literature [35-37]. Therefore, normative task data were stratified by age. Hearing sensitivity and sensitivity to mistuned, oddball, or discontinuous tones have been shown to deteriorate with age [38-40]. The slowing of processing speed with age has also been well documented [41,42]. Visuospatial working memory performance also decreases with age, potentially because of differences in chunking strategies and proactive interference [43-45]. Better performance observed with path finder, a task of visuospatial learning, may be due in part to more efficient within-task adaptability by younger individuals, a result identified previously with visuospatial tasks [43]. Other similar age-related associations have been documented in learning paradigms [46]. The findings of this study of decreased cognitive flexibility with age are also largely supported by other studies, such as those examining task-switching capacity [47,48]. Finally, in the literature, older age has been found to be related to poorer emotion recognition, processing facial emotion, and inhibitory control [49-51]. These results likely explain why performance on tap the emotion and face to face, tasks of emotion processing and inhibitory control, was negatively associated with age; however, it is unclear why emotional face did not show an association. Given these findings, which are supported by the greater cognitive literature, normative task data were stratified by age. Therefore, normative task data were stratified by age.

Better performance on a visual learning task was observed in men, corroborating previous gender findings with learning tasks that involve spatial navigation and manipulation [52,53]. These results appear to follow previous findings, suggesting male advantage in detecting interaural time and intensity differences, complex masking tasks, and lateralization of auditory discrimination processing, although these differences are small [54,55]. With samples larger than those in this study, age- and gender-specific norms could be constructed to better characterize the individual performance.

Although previous research has demonstrated associations between performance on cognition and education tests, our task battery did not show relationships with GPA or years of education after controlling for age [56,57]. These findings may indicate that the ONAs successfully captured neural system functioning as opposed to more notion-based intelligence. In addition, only bubble pop performance was related to the occupation level after controlling for age, suggesting a potential benefit of work status.

The majority of predominantly visual ONAs were only minimally associated with one another, as expected, suggesting that they are likely to tap into unique domains of performance. Beep seeker and bubble pop, which measure sensory memory and working memory across auditory and visuospatial domains, respectively, were related. Previous research has supported this association and provided evidence for similar neural mechanisms underlying both memory systems [58,59]. In addition, 2 of the 3 tasks of emotion processing were only minimally correlated with one another. This finding is expected given the varying demands on processes of inhibition between the tasks and differences in the facial targets (eg, responding to faces with any emotion vs identifying faces with matching emotions). It is surprising that tap the emotion and emotional face were not associated, given that both tasks involved processing of facial emotions while inhibiting a prepotent response, although it is unclear whether this relationship would exist with a larger sample who performed both tasks.

Despite the recent development of various digital cognitive assessment tools, few studies have released normative data sets, thus limiting the information that can be gained from these tasks. Collecting normative data from the general community is an important and necessary first step before interpreting cognitive performance in clinical populations. The current global health crisis has arrested the ability to administer traditional in-person cognitive assessments in clinical and research settings. The development of valid and reliable assessment platforms, which can be delivered remotely, has become crucial.

Despite the strengths of our results, a few limitations exist that should be considered. Although 74.1% (579/781) of the sample came from the general community, a subset of participants was recruited through their attendance at a large public college institution. Therefore, young adults’ performance in this sample may differ somewhat from that of the general population. Due to the limited overlap of measures that each participant completed, we were unable to evaluate whether subsets of the tasks were explained by common underlying factors to investigate construct validity. Similarly, the correlations between tasks should be interpreted with caution, given the variability in the number of individuals who were administered each task. Data on participant race and ethnicity were also not collected as part of this study. In addition, 2 individuals identified as intersex prevented our ability to reliably predict how a greater population of intersex individuals may perform on the presented tasks. Future work should aim to evaluate race and gender minorities more specifically to better understand the performance and feasibility of these populations.

Experiential cognitive assessment in patients’ homes may help gain a better understanding of their true cognitive states, but it also raises the possibility that individuals will lack the attention or motivation to properly engage in testing [9]. Failure to prevent some patients from using substances or eliciting outside help during testing may also be a necessary aspect of remote testing [24,60]. In these situations, performance on a given task may not truly represent the intended cognitive domain, as it is likely to be in a distraction-limited environment. More information is also needed to understand how cognition in the general community fluctuates from shorter to more extended periods when assessed remotely.

The accessibility of these and other digital assessments provides an opportunity to integrate objective behavioral assessments directly into medical records to guide care. Given that the field of health informatics is still budding, future work should evaluate the extent to which these tasks capture unique variance and predict outcomes amidst other data.

This study presented the performance of individuals from the general community using a novel cognitive assessment tool that can be employed remotely. Participants found the tasks to be interactive and easy to use. The development and validation of web-based tasks such as these widely expand the accessibility of cognitive assessment to new populations such as rural groups and others with limited physical mobility and may also increase the ability of health practitioners to conduct repeated testing. Quick, easy-to-use digital assessment platforms with remote capabilities such as this may help bring the field closer to achieving more impactful patient-centered care.