Published on in Vol 24, No 1 (2022): January

Preprints (earlier versions) of this paper are available at, first published .
A Novel Virtual Reality Assessment of Functional Cognition: Validation Study

A Novel Virtual Reality Assessment of Functional Cognition: Validation Study

A Novel Virtual Reality Assessment of Functional Cognition: Validation Study

Original Paper

1Institute of Psychiatry, Psychology & Neuroscience, King's College London, London, United Kingdom

2Trafford Centre for Medical Research, University of Sussex, Brighton, United Kingdom

3Faculty of Media and Communications, Bournemouth University, Bournemouth, United Kingdom

4School of Psychology, University of East London, London, United Kingdom

5Kent and Medway Medical School, Canterbuy, United Kingdom

6Kent and Medway National Heath Service and Social Care Partnership Trust, Gillingham, United Kingdom

*these authors contributed equally

Corresponding Author:

Lilla Alexandra Porffy, BSc, MSc

Institute of Psychiatry, Psychology & Neuroscience, King's College London

16 De Crespigny Park

London, SE5 8AF

United Kingdom

Phone: 44 07738244217


Background: Cognitive deficits are present in several neuropsychiatric disorders, including Alzheimer disease, schizophrenia, and depression. Assessments used to measure cognition in these disorders are time-consuming, burdensome, and have low ecological validity. To address these limitations, we developed a novel virtual reality shopping task—VStore.

Objective: This study aims to establish the construct validity of VStore in relation to the established computerized cognitive battery, Cogstate, and explore its sensitivity to age-related cognitive decline.

Methods: A total of 142 healthy volunteers aged 20-79 years participated in the study. The main VStore outcomes included verbal recall of 12 grocery items, time to collect items, time to select items on a self-checkout machine, time to make the payment, time to order coffee, and total completion time. Construct validity was examined through a series of backward elimination regression models to establish which Cogstate tasks, measuring attention, processing speed, verbal and visual learning, working memory, executive function, and paired associate learning, in addition to age and technological familiarity, best predicted VStore performance. In addition, 2 ridge regression and 2 logistic regression models supplemented with receiver operating characteristic curves were built, with VStore outcomes in the first model and Cogstate outcomes in the second model entered as predictors of age and age cohorts, respectively.

Results: Overall VStore performance, as indexed by the total time spent completing the task, was best explained by Cogstate tasks measuring attention, working memory, paired associate learning, and age and technological familiarity, accounting for 47% of the variance. In addition, with λ=5.16, the ridge regression model selected 5 parameters for VStore when predicting age (mean squared error 185.80, SE 19.34), and with λ=9.49 for Cogstate, the model selected all 8 tasks (mean squared error 226.80, SE 23.48). Finally, VStore was found to be highly sensitive (87%) and specific (91.7%) to age cohorts, with 94.6% of the area under the receiver operating characteristic curve.

Conclusions: Our findings suggest that VStore is a promising assessment that engages standard cognitive domains and is sensitive to age-related cognitive decline.

J Med Internet Res 2022;24(1):e27641




Cognitive dysfunction refers to deficits in intellectual functions usually described by domains such as attention, working memory, verbal and visual learning, executive function, and processing speed. Deficits in cognition are evident across a range of neuropsychiatric disorders, including Alzheimer disease (AD), schizophrenia, and depression. Although these intellectual deficits are diagnostic in AD [1], 90% of individuals with schizophrenia and depression are also affected [2,3]. This is further complicated by the observation that some cognitive decline is part of the natural aging process and is reported in one-quarter of older adults without dementia [4]. The high prevalence of cognitive dysfunction in mental and physical [5] illness, an increasingly aging population, and the lack of robust treatments suggest that the global burden of cognitive dysfunction has a substantial socioeconomic impact.

Cognitive decline has a marked effect on functional recovery and quality of life in patients with mental disorders [6-8]. In addition, it precedes and predicts functional outcomes in AD and schizophrenia [9,10] and predicts treatment response in depression [11], highlighting the urgent need to effectively target these symptoms. Unfortunately, clinical trials of cognitive enhancers have been largely disappointing [12-14]. Indeed, most compounds that demonstrated positive effects in phase 2 trials have failed in phase 3 trials. This raises several questions about the sensitivity of our cognitive assessments and the targets of these interventions.

Standard cognitive assessments are designed to evaluate changes in distinct neuropsychological domains, whereas the actual target for therapy is change in functional cognition—the ability to perform everyday routine activities [15]. Accordingly, the Food and Drug Administration has mandated the assessment of real-life functional change, alongside changes in conventional cognitive performance, as a condition for drug approval for both AD and schizophrenia [16,17]. This is particularly important as there can be a lack of concordance between changes in cognitive measures and related everyday functioning. For example, cognitive task performance only explains 20% of the variance in work-related skills in schizophrenia [18]. Although there has been an attempt to supplement cognitive assessments with self-report and reports by caregivers to assess wider functioning, these assessments lack objectivity.

A related issue is that cognitive assessments require optimal task engagement, which can be confounded by poor attention and motivation [19,20]. The gold standard cognitive measure for AD, the Alzheimer’s Disease Assessment Scale–Cognitive Subscale [21], takes approximately 45 minutes to administer; the analogous scale for schizophrenia, the MATRICS Consensus Cognitive Battery [22,23], takes up to 90 minutes. The related functional capacity assessment for AD, the Clinical Dementia Rating Scale [24], takes approximately 30 minutes to complete, similar to the University of California, San Diego Performance-Based Skills Assessment [25] for schizophrenia. The ecological validity and predictive power of real-life performance of standard assessments have also been questioned [26,27].

Complex assessments that emulate everyday scenarios have been developed, including the Multiple Errands Test (MET), which measures executive function in patients with traumatic brain injury [28], and the Virtual Reality Functional Capacity Assessment Tool (VRFCAT), which measures functional skills in schizophrenia and can reliably differentiate patients from controls [29,30]. However, the MET is time-consuming and difficult to standardize with a lack of experimental control, whereas the VRFCAT lacks full ecological validity as it is completed on a computer or tablet without the immersive nature of real-life interactions.


Recent developments in technology, specifically in virtual reality (VR), now enable us to create assessments that can replicate challenges found in everyday life while also maintaining experimental control [31]. This offers the opportunity to overcome issues associated with current assessments. In this study, we describe the development of a novel, fully immersive VR assessment, VStore, with the aim to simultaneously assess traditional cognitive domains and functional capacity. This is achieved through the creation of an ecologically valid minimarket environment with a maze-like layout. Each action within the assessment maps an embedded cognitive task (eg, recall of shopping list items measures verbal memory), and each task is assessed by performing actions that require almost identical procedures similar to shopping in real life, offering a measure of concurrent functional capacity. Moving in an immersive VR environment engages brain structures associated with spatial navigation, such as the hippocampus and entorhinal cortex [32], which are affected in early AD [33], depression [34], and schizophrenia [35]. Therefore, VStore may be more sensitive to early neurodegenerative processes than the existing assessments.

The aim of this study is 2-fold. First, we establish the cognitive domains relevant to VStore performance. More specifically, we test which cognitive processes, as measured by an existing standard cognitive battery, predict VStore performance as an initial evaluation of its construct. We achieve this by conducting a series of stepwise prediction models. Second, we explore the preliminary utility of VStore in assessing cognitive decline associated with nonpathological aging. This is achieved by testing VStore’s ability to predict age both as a continuous and dichotomized outcome.


A total of 142 healthy volunteers aged 20-79 years were recruited through advertisements in college circular emails, charity newsletters, and social media. Participants were excluded if they had (1) a diagnosis of an axis 1 disorder (Diagnostic and Statistical Manual of Mental Disorders, 5th edition); (2) dependence on alcohol or illicit substances; (3) clinically significant motion sickness; (4) a pregnancy; and (5) a diagnosis of a neurological illness. Of the 142 volunteers, 38 (26.8%) participants were excluded from the study. The reasons for exclusion were as follows: 1 participant withdrew consent, 1 could not complete VStore owing to technical issues, 1 senior participant could not complete VStore owing to fatigue, 20 participants failed either or both integrity and completion criteria for Cogstate, and 15 participants were removed owing to outlier values on one or more primary outcome measures. The demographic information for the final sample of 73.2% (104/142) of participants is presented in Table 1.

Table 1. Sample demographics.
VariableAge group (years)

Population, n (%)19 (18.3)18 (17.3)18 (17.3)17 (16.3)18 (17.3)14 (13.5)104 (100)
Age (years), mean (SD; range)23.6 (2.5; 20-29)32.7 (2.3; 30-37)45.2 (2.8; 40-49)53.5 (3.0; 50-59)64.2 (2.4; 61-69)73.4 (3.3; 70-79)48.8 (17.8; 20-79)
Gender (female), n (%)10 (52.6)11 (61.1)9 (50)8 (47.1)10 (55.6)6 (42.9)54 (51.9)
IQ, mean (SD; range)119.7 (9.5; 96-136)121.3 (6.9; 105-131)120.6 (6.0; 106-132)121.7 (7.9; 109- 133)126.3 (5.9; 113-134)128.1 (6.3; 118-138)122.9 (10.3; 96-138)
Education (years); mean (SD; range)17.6 (2.4; 15-24)19.4 (3.0; 14-27)18.8 (1.9; 15-22)18.0 (9.5; 11-20)16.1 (3.4; 10-23)18.1 (5.3; 10-25)18.0 (3.6; 10-27)
Technological familiarity, mean (SD; range)47.1 (6.8; 34-57)47.2 (6.4; 32-57)43.5 (7.9; 32-58)43.4 (7.9; 28-57)35.6 (8.3; 21-52)33.4 (8.6; 21-48) 41.7 (9.7; 21-58)



VStore was developed in collaboration with Vitae VR [36]. It takes approximately 30 minutes to complete, including orientation, instructions, practice, and assessment. Orientation and practice are set in a courtyard specifically designed for VR acclimatization (Multimedia Appendix 1).

The assessment itself is set in a minimarket environment depicting a fruit and vegetable section; 6 aisles of foodstuff, snacks, drinks, and toiletries; fridges with chilled drinks and sandwiches; and freezers with frozen meals. In addition, there are checkout and self-checkout counters and a coffee shop at the back of the minimarket. A total of 66 items, organized into 9 categories, were created to fill the shop (Multimedia Appendix 2).

At the start, participants were read out 12 items from a shopping list (Multimedia Appendix 3) by the avatar standing near the entrance. The first task of the participants was to memorize and recall as many items from this list as possible. Following recall, participants were presented with the shopping list, including all 12 items, and instructed to move around the shop and collect all items as quickly and accurately as possible. Once all the items are bagged, they are required to select and pay for them at a self-checkout machine, providing the exact amount (Multimedia Appendix 4). The task is concluded when participants order a hot drink from the coffee shop situated in the minimarket. Progression to the next task could only be achieved after successfully completing the previous task. The steps required to complete the VStore tasks are summarized in Figure 1. Multimedia Appendices 5-7 provide details on apparatus information, software information, and how movement is executed in the virtual environment, respectively.

Figure 1. Flowchart depicting the steps required to complete VStore, its corresponding cognitive domains, and outcome variables.
View this figure

Cogstate is a computerized cognitive battery designed to assess multiple cognitive domains. It has been widely used in both healthy and clinical populations. Cogstate is simple to use, even for adults with limited computer experience, and therefore suitable for testing older adults [37]. For the purposes of this study, 8 tasks that cover key cognitive domains (Table 2 and Multimedia Appendix 8) were selected, taking approximately 30-40 minutes to complete.

Table 2. List of Cogstate tasks, corresponding cognitive domains assessed, and main outcome measures.
CodeTask nameCognitive domainOutcome (metric)
DETDetectionProcessing speedReaction time (log 10 ms)
IDNIdentificationAttentionReaction time (log 10 ms)
OCLOne Card LearningVisual learningAccuracy (arcsine proportion)
ONBOne-BackWorking memoryReaction time (log 10 ms)
TWOTwo-BackWorking memoryAccuracy (arcsine proportion)
CPALContinuous Paired Associate LearningPaired associate learningTotal number of errors (N/Aa)
GMLTGroton Maze LearningExecutive functionsTotal number of errors (N/A)
ISLTInternational Shopping List TaskVerbal learning Number of correct responses (N/A)

aN/A: not applicable.

Wechsler Abbreviated Scale of Intelligence

The abbreviated version of the Wechsler Adult Intelligence Scale was used to establish the IQ of participants [38]. Specifically, the 2-item scale included matrix and vocabulary tests.

Technological Familiarity Questionnaire

We developed a self-report questionnaire to assess the technological familiarity of the sample population. Participants were asked 13 questions to ascertain their frequency, comfort, and ability in technology use. Higher scores indicated more technological familiarity. The internal consistency of the questionnaire was good (Cronbach α=.88). A detailed description of the Technological Familiarity Questionnaire (TFQ) is presented in Multimedia Appendix 9.


Potential participants were prescreened over the phone. If they were deemed eligible, they were invited for a single study visit that lasted up to 2.5 hours. First, informed consent was obtained, followed by obtaining demographics, brief mental and physical health history, and the TFQ scores. Cogstate and VStore were administered in a counterbalanced fashion to mitigate any order effects. All the participants received the same shopping list. Finally, the Wechsler Abbreviated Scale of Intelligence was administered. Participants were compensated for their time and reimbursed for travel expenses. Ethical approval was granted by the Psychiatry, Nursing and Midwifery Research Ethics Committee, King’s College London (LRS-16/17-4540).


Before data analysis, VStore outcome variables measured in seconds were log-transformed to stabilize the variance. Descriptive statistics for both VStore and Cogstate outcomes are presented in Multimedia Appendices 10 and 11. As an initial overview of the relationship between Cogstate and VStore, Bonferroni-corrected Spearman ρ was calculated between the 2 assessments. These results are presented in Multimedia Appendix 12.

To establish which cognitive domains, assessed by Cogstate, best predicted VStore performance, we ran a series of backward elimination regression models implemented in the R package MASS [39]. VStore outcomes were entered as dependent variables (DVs) and all 8 Cogstate tasks were entered as independent variables (IVs). Age and technological familiarity (TFQ) were also entered as IVs, as these (but not IQ) showed a significant relationship with VStore outcomes. All IVs were standardized using the sample mean and SD to create z scores. Regression models were penalized for complexity using the Akaike Information Criterion (AIC) to arrive at the most parsimonious model. Additional quality checks for the final models are presented in Multimedia Appendix 13. These confirm that the assumptions of normality and homoscedasticity were met.

As an exploratory objective to examine the potential of VStore in predicting age, we used ridge regression, implemented in the R package glmnet [40], where regularization is governed by 2 parameters— α and λ. We set the penalty parameter, α, to 0 (to enforce ridge regression, where the estimated coefficients of strongly correlated predictor variables are shrunk toward each other). The optimal value of the strength of this penalty (λ) was determined using leave-one-out cross-validation (ie, for a given value of λ, training on N-1 participants, and testing performance on the one participant who is held-out by computing the mean squared error [MSE]). The DV was age for 104 participants. In the first model, IVs included all VStore outcomes except for total time: Recall, Find, Select, Pay, and Coffee. In the second model, IVs included all Cogstate tasks: Detection (DET), Identification (IDN), One Card Learning (OCL), One-Back (ONB), Two-Back (TWO), Continuous Paired Associate Learning (CPAL), Groton Maze Learning (GMLT), and the International Shopping List Task (ISLT). Both models were repeated with technological familiarity (TFQ) included as an additional IV to indicate whether VStore was confounded by technological familiarity. Finally, to further probe VStore’s sensitivity in predicting age cohorts, we took the top and bottom 20% of the sample population based on age and ran 2 logistic regression models to generate 2 overlying receiver operating characteristic curves—one for VStore and one for Cogstate. The bottom fraction of the sample included 23.1% (24/104) of participants aged 20-30 years, whereas the top fraction included 22.1% (23/104) of participants aged 65-79 years. Similar to the regression analyses, the age cohort (0, 1) was entered as the DV, and IVs for VStore model were Recall, Find, Select, Pay, and Coffee, whereas the IVs for the Cogstate model included DET, IDN, OCL, ONB, TWO, CPAL, GMLT, and ISLT. Youden J statistic was used to establish the optimal threshold for sensitivity and specificity, and model performance was compared with the DeLong test.

VStore Construct

Tables 3-5 summarizes the predictors of VStore performance. The initial model included all Cogstate variables, in addition to age and technological familiarity. Backward elimination regression resulted in the removal of several of these predictors, without any substantial change in the variance explained by the models. AIC values showed a decrease from the initial to final models, arriving at a more parsimonious set of predictors for each VStore outcome.

Recalling items from VStore shopping list was predicted by verbal learning. Finding items in VStore was best explained by attention (IDN), working memory (ONB), paired associate learning (CPAL), age, and technological familiarity (TFQ). The best predictors of VStore Select were working memory (TWO), executive functions (GMLT), verbal learning (ISLT), and age. Paying for items in VStore was best explained by processing speed (DET), working memory (TWO), executive function (GMLT), verbal learning (ISLT), and technological familiarity (TFQ). Time to order a coffee was best predicted by visual (OCL) and verbal (ISLT) learning, working memory (TWO), and age. Finally, total time spent in VStore was best explained by attention (IDN), working memory (ONB), paired associate learning (CPAL), age, and technological familiarity (TFQ). For the final model, the explained variance ranged from 25% for VStore Select to 47% for VStore Total time.

Given the prominent role of technological familiarity in VStore performance, we also examined the correlations between the TFQ and Cogstate for comparison. Indeed, 6 out of 8 Cogstate tasks (DET, IDN, ONB, TWO, CPAL, and GMLT) had a significant relationship with the TFQ (Multimedia Appendix 14).

Table 3. Initial and final linear regression models examining the construct validity of VStore for Recall, Find, and Select outcomes.
DVa and IVbInitial modelFinal model

B (SE)P valueR2F test (df)AICcB (SE)P valueR2F test (df)AIC

DETd0.264 (0.205).20N/AeN/AN/AN/AN/AN/AN/AN/A

IDNf−0.064 (0.224).78N/AN/AN/AN/AN/AN/AN/AN/A

OCLg0.078 (0.190).68N/AN/AN/AN/AN/AN/AN/AN/A

ONBh−0.016 (0.520).95N/AN/AN/AN/AN/AN/AN/AN/A

TWOi0.246 (0.218).26N/AN/AN/AN/AN/AN/AN/AN/A

GMLTj−0.056 (0.214).79N/AN/AN/AN/AN/AN/AN/AN/A

CPALk0.209 (0.214).33N/AN/AN/AN/AN/AN/AN/AN/A

ISLTl0.569 (0.202).006N/AN/AN/A0.056 (0.014)<.001N/AN/AN/A

Age−0.312 (0.242).20N/AN/AN/AN/AN/AN/AN/AN/A

TFQm0.001 (0.228).95.102.138 (10, 93)123.6N/AN/A.1316.210n (1, 102)111.8

DET0.019 (0.028).047N/AN/AN/AN/AN/AN/AN/AN/A

IDN−0.061 (0.030).93N/AN/AN/A−0.052 (0.028).07N/AN/AN/A

OCL−0.002 (0.026).06N/AN/AN/AN/AN/AN/AN/AN/A

ONB0.065 (0.034).92N/AN/AN/A0.074 (0.030).03N/AN/AN/A

TWO−0.003 (0.029).40N/AN/AN/AN/AN/AN/AN/AN/A

GMLT0.024 (0.029).11N/AN/AN/AN/AN/AN/AN/AN/A

CPAL0.046 (0.029).84N/AN/AN/A0.055 (0.025).03N/AN/AN/A

ISLT−0.006 (0.027).006N/AN/AN/AN/AN/AN/AN/AN/A

Age0.093 (0.033).03N/AN/AN/A0.100 (0.029).001N/AN/AN/A

TFQ−0.070 (0.031).50.407.833n (10, 93)−293.1−0.068 (0.028).02.4215.830n (5, 98)−301.1

DET−0.006 (0.034).86N/AN/AN/AN/AN/AN/AN/AN/A

IDN−0.035 (0.037).35N/AN/AN/AN/AN/AN/AN/AN/A

OCL−0.009 (0.031).77N/AN/AN/AN/AN/AN/AN/AN/A

ONB0.020 (0.041).64N/AN/AN/AN/AN/AN/AN/AN/A

TWO−0.034 (0.036).35N/AN/AN/A−0.043 (0.030).16N/AN/AN/A

GMLT0.041 (0.035).252N/AN/AN/A0.043 (0.031).16N/AN/AN/A

CPAL0.024 (0.035).50N/AN/AN/AN/AN/AN/AN/AN/A

ISLT−0.041 (0.033).22N/AN/AN/A−0.047 (0.030).12N/AN/AN/A

Age0.111 (0.040).007N/AN/AN/A0.108 (0.030).001N/AN/AN/A

TFQ0.001 (0.038).97.223.896n (10, 93)−251.6N/AN/A.259.783n (4, 99)−261.8

aDV: dependent variable.

bIV: independent variable.

cAIC: Akaike Information Criterion.

dDET: Detection (processing speed).

eN/A: not applicable.

fIDN: Identification (attention).

gOCL: One Card Learning (visual learning).

hONB: One-Back (working memory).

iTWO: Two-Back (working memory).

jGMLT: Groton Maze Learning (executive function).

kCPAL: Continuous Paired Associate Learning (paired associate learning).

lISLT: International Shopping List Task (Verbal learning).

mTFQ: Technological Familiarity Questionnaire.

nSignificant at P<.001.

Table 4. Initial and final linear regression models examining the construct validity of VStore for Pay and Coffee outcomes.
DVa and IVbInitial modelFinal model

B (SE)P valueR2F test (df)AICcB (SE)P valueR2F test (df)AIC

DETd0.043 (0.040).28N/AeN/AN/A0.063 (0.035).08N/AN/AN/A

IDNf0.024 (0.043).59N/AN/AN/AN/AN/AN/AN/AN/A

OCLg0.032 (0.037).38N/AN/AN/AN/AN/AN/AN/AN/A

ONBh0.034 (0.048).49N/AN/AN/AN/AN/AN/AN/AN/A

TWOi−0.085 (0.042).047N/AN/AN/A−0.085 (0.036).02N/AN/AN/A

GMLTj0.070 (0.041).09N/AN/AN/A0.060 (0.037).11N/AN/AN/A

CPALk−0.025 (0.041).55N/AN/AN/AN/AN/AN/AN/AN/A

ISLTl−0.072 (0.039).07N/AN/AN/A−0.077 (0.034).03N/AN/AN/A

Age0.020 (0.047).67N/AN/AN/AN/AN/AN/AN/AN/A

TFQm−0.106 (0.044).02.346.409n (10, 93)−218.6−0.129 (0.035)<.001.3612.630n (5, 98)−225.8

DET0.018 (0.048).72N/AN/AN/AN/AN/AN/AN/AN/A

IDN−0.018 (0.053).73N/AN/AN/AN/AN/AN/AN/AN/A

OCL0.075 (0.045).10N/AN/AN/A0.077 (0.041).07N/AN/AN/A

ONB0.021 (0.059).72N/AN/AN/AN/AN/AN/AN/AN/A

TWO−0.046 (0.051).37N/AN/AN/A−0.078 (0.042).07N/AN/AN/A

GMLT−0.003 (0.051).96N/AN/AN/AN/AN/AN/AN/AN/A

CPAL0.061 (0.051).23N/AN/AN/AN/AN/AN/AN/AN/A

ISLT−0.060 (0.048).21N/AN/AN/A−0.066 (0.044).13N/AN/AN/A

Age0.181 (0.057).002N/AN/AN/A0.218 (0.043)<.001N/AN/AN/A

TFQ−0.027 (0.054).62.274.751n (10, 93)−177.0N/AN/A.3011.810n (4, 99)−186.6

aDV: dependent variable.

bIV: independent variable.

cAIC: Akaike Information Criterion.

dDET: Detection (processing speed).

eN/A: not applicable.

fIDN: Identification (attention).

gOCL: One Card Learning (visual learning).

hONB: One-Back (working memory).

iTWO: Two-Back (working memory).

jGMLT: Groton Maze Learning (executive function).

kCPAL: Continuous Paired Associate Learning (paired associate learning).

lISLT: International Shopping List Task (Verbal learning).

mTFQ: Technological Familiarity Questionnaire.

nSignificant at P<.001.

Table 5. Initial and final linear regression models examining the construct validity of VStore Total.
DVa and IVbInitial modelFinal model

B (SE)P valueR2F test (df)AICcB (SE)P valueR2F test (df)AIC

DETd0.018 (0.020).44N/AeN/AN/AN/AN/AN/AN/AN/A

IDNf−0.046 (0.024).08N/AN/AN/A−0.039 (0.025).12N/AN/AN/A

OCLg0.003 (0.026).88N/AN/AN/AN/AN/AN/AN/AN/A

ONBh0.048 (0.022).10N/AN/AN/A0.067 (0.026).01N/AN/AN/A

TWOi−0.015 (0.029).56N/AN/AN/AN/AN/AN/AN/AN/A

GMLTj0.025 (0.025).32N/AN/AN/AN/AN/AN/AN/AN/A

CPALk0.043 (0.025).09N/AN/AN/A0.059 (0.021).007N/AN/AN/A

ISLTl−0.023 (0.025).34N/AN/AN/AN/AN/AN/AN/AN/A

Age0.097 (0.024).001N/AN/AN/A0.109 (0.025)<.001N/AN/AN/A

TFQm−0.050 (0.028).06.469.803n (10, 93)−323.3−0.045 (0.025).07.4719.070n (5, 98)−329.1

aDV: dependent variable.

bIV: independent variable.

cAIC: Akaike Information Criterion.

dDET: Detection (processing speed).

eN/A: not applicable.

fIDN: Identification (attention).

gOCL: One Card Learning (visual learning).

hONB: One-Back (working memory).

iTWO: Two-Back (working memory).

jGMLT: Groton Maze Learning (executive function).

kCPAL: Continuous Paired Associate Learning (paired associate learning).

lISLT: International Shopping List Task (Verbal learning).

mTFQ: Technological Familiarity Questionnaire.

nSignificant at P<.001.

Cognitive Performance as Predictor of Age

For the DV age, we built 2 models using VStore and Cogstate outcomes as predictors (Figure 2). In the VStore model, the model fitting achieved an MSE of 185.8 (SE 19.34), selecting a total of 5 predictors and from cross-validating, which was attained from an optimal λ of 5.16 (Multimedia Appendix 15). For the Cogstate model, we found an MSE of 226.8 (SE 23.48), selecting a total of 8 predictors obtained with an optimal λ of 9.49 (Multimedia Appendix 16). We also fitted a null (intercept-only) model that yields an MSE of 294.71, suggesting that models for both VStore and Cogstate are preferable to a model with no predictors.

Figure 2. VStore and Cogstate models predicting age. Dashed lines depict the age of the participants. The solid stroke shows the age predicted by the ridge regression models.
View this figure

In the VStore model, coefficient values were as follows: VStore Recall=−0.586; VStore Find=7.882; VStore Select=5.284; VStore Pay=3.291; and VStore Coffee=4.526. In this model, the Find task is most strongly positively associated with increasing age, followed by the Select, Coffee, Pay, and Recall tasks.

In the Cogstate model, the coefficient values were as follows: DET=11.089; IDN=15.277; OCL=−2.563; ONB=12.038; TWO=−1.293; GMLT=0.032; CPAL=0.015; and ISLT=−0.245. In this model, the IDN task was most strongly positively associated with increasing age, followed by the ONB, DET, and TWO tasks.

For the DV age, we built 2 additional models using VStore and Cogstate outcomes as predictors with technological familiarity included as a covariate (Figure 3). With the TFQ added to the VStore model, the model fitting achieved an MSE of 162.9 (SE 17.50), selecting a total of 6 predictors and from cross-validating, this was attained from an optimal λ at 3.904 (Multimedia Appendix 17). With the TFQ added to the Cogstate model, we found an MSE of 175.4 (SE 22.12), selecting a total of 9 predictors obtained with an optimal λ at 2.904 (Multimedia Appendix 18).

Figure 3. VStore and Cogstate models predicting age with technological familiarity included. Dashed lines depict the age of the participants. The solid stroke shows the age predicted by the ridge regression models.
View this figure

In the VStore model, coefficient values were as follows: VStore Recall=−0.617; VStore Find=7.298; VStore Select=5.422; VStore Pay=2.699; VStore Coffee=4.517, and TFQ=−0.279.

In the Cogstate model, coefficient values were as follows: DET=16.777; IDN=22.388; OCL=−3.544; ONB=15.451; TWO=2.034; GMLT=0.0438; CPAL=0.0243, ISLT=−0.479; and TFQ=−0.348.

Age Cohort Classification

Figure 4 shows the sensitivity of the VStore and Cogstate models in classifying age cohorts of 20-30 and 65-79 years of this study’s sample population. VStore has a sensitivity of 87% and specificity of 91.7% at the optimal threshold of 0.55, whereas Cogstate has a sensitivity of 95.7% and specificity of 75% at the optimal threshold of 0.36. The difference between the 2 models was not statistically significant (Z=0.69, P=.49).

Figure 4. VStore and Cogstate models predicting age cohorts. AUC: area under the receiver operating characteristic curve.
View this figure

Principal Findings

The primary aim of this study was to establish which cognitive functions are engaged during a novel VR assessment, VStore. We found that a number of cognitive processes, as measured by Cogstate, contributed to the variance explained in VStore performance, suggesting that the VR task engages a range of key neuropsychological functions simultaneously. Indeed, the realistic nature of VStore precludes a simple one-to-one mapping between Cogstate domains and VStore outcomes. These findings provide preliminary information about VStore’s construct validity and show that functional tasks embedded in VR may engage a greater range of cognitive domains than standard assessments because of their increased complexity and ability to resemble the demands of the real world [41].

As anticipated, VStore Recall was best explained by the Cogstate verbal learning task. VStore Find demonstrated a significant relationship with a number of predictors including attention, working memory, paired associate learning, age, and technological familiarity. VStore Select was explained by working memory, executive function, verbal learning, and technological familiarity. The more items participants could remember, the quicker they selected them on the self-checkout machine (verbal learning); attentional control (executive function) and temporary memorization of remaining items (working memory) were also required. VStore Pay engaged working memory, executive functions, and required processing speed. VStore Coffee was explained by visual and verbal learning, working memory, and age. Finally, the total time spent in VStore was best explained by Cogstate tasks measuring paired associate learning and working memory, in addition to the participants’ age and technological familiarity, accounting for almost half of the variance in VStore performance.

The CPAL task of Cogstate is an episodic memory paradigm that involves visuospatial processing and indexes the ability to learn, store, and retrieve information. Paired associations may be especially important when finding items in a store, as this requires the retrieval of object representations from the shopping list, such as Cornflakes. Severe impairment in this domain has been linked to a number of neuropsychiatric conditions, including AD [42], and has been shown to be a valuable tool for the early detection of the disorder [43]. Deficits in paired associate learning have also been observed in schizophrenia and are linked to hippocampal volume loss [44].

Working memory, the temporary retention of information for manipulation and decision-making, is a key cognitive process in overall VStore performance. It is particularly relevant for the stages of the assessment where reviewing the shopping list is necessary to successfully carry out the next step of the task, such as finding an item or selecting it on the self-checkout machine. In support of the role of working memory in complex cognitive and functional assessments, factor analysis revealed that working memory was one of the latent variables of the VRFCAT, among problem solving and processing speed [45]. A decline in working memory has been reported in both AD and schizophrenia [46,47]. Working memory also declines as part of the normal aging process [48].

The ability of VStore to engage cognitive domains implicated in neuropsychiatric disorders and age-related cognitive decline points to its potential in assessing functional cognition not only in healthy individuals but also in clinical populations. In this study, the total time spent in VStore increases with age; hence, age is a significant predictor of most VStore outcomes. However, this may partly be attributed to decrease in technological familiarity with age [49], which could also play a significant role in the outcome of digital assessments. Indeed, ridge regression revealed that the main VStore outcomes—Recall, Find, Select, Pay, and Coffee—provide a parsimonious model and can predict age accurately. Although we cannot make a direct comparison between VStore and Cogstate models, it is observed that Cogstate has a larger slope deviation from the identity line than VStore. Intriguingly, although the inclusion of technological familiarity made VStore model less precise, it did not alter the overall results. In contrast, the Cogstate model was markedly improved by the addition of technological familiarity. This may be because of the additional technological demands of the VStore setup, despite the intuitive nature of the task. The fact that the addition of technological familiarity did not improve the VStore model could be because the variance associated with technological skills was already captured, whereas for Cogstate, this was not the case. As technological familiarity decreased with age, we cannot rule out that VStore, similar to any other digital assessment, may potentially underestimate the cognitive abilities of older adults. As VR tools become more familiar, this relationship may reduce over time, and thus we recommend the assessment of technological familiarity in studies that include participants where these skills may vary.

Similar to these findings, receiver operating characteristic curve analysis revealed that VStore is highly accurate, sensitive, and specific to the classification of age cohorts, further supporting its potential use in the assessment of age-related cognitive decline. This is in line with previous research showing that age is a relevant factor in performance on VR assessments [49,50], potentially explicable by the decline in exploratory navigational abilities—a domain particularly vulnerable to the effects of aging [51]. Effective exploration and navigation are vital for completing VStore and are likely to engage relevant brain regions. Indeed, a key aim in designing the VStore Find task was to activate the place and grid cells in the hippocampus and entorhinal cortex [52]. Notably, this variable was the most strongly associated with increasing age, suggesting that spatial processing, as assessed by VStore, could be used to inform future normative data to detect below-average performance for specified age brackets with high sensitivity.


There are several limitations to this study. First, the study sample had a high IQ on average, as expected from our highly educated cohort; hence, the sample may not be fully representative of the general population. This may be due to an oversampling from college students and a better-educated general population, and the use of the abbreviated IQ measure that relies on only 2 domains, verbal ability and matrix reasoning, and may generate inflated scores [53]. Nonetheless, we were able to include a range of IQ scores. Furthermore, relying on the AIC stepwise algorithm for model selection is not ideal, as it may be affected by several factors, such as the degree of correlation between predictors or the size of the sample, and thus may not be fully replicable [54]. Although theory-based model selection is preferable, given the novelty of the VR task, this was not possible on this occasion. In addition, although ridge regression models were cross-validated by optimizing λ, these models should be validated in an independent sample. Future research should also include measures of adverse VR effects; however, it is important to note here that no participant stopped the VR assessment because of cybersickness. Similarly, although there has been no functional capacity assessment developed for healthy adults, the inclusion of a proxy measure, such as the Cognitive Failures Questionnaire [55], would have been desirable. Finally, further research is required to confirm the construct validity of VStore and, most importantly, establish its test-retest reliability.


In conclusion, our findings suggest that VStore is a promising assessment that engages various cognitive functions, including those that tend to decline with age and during the development of neuropsychiatric disorders such as AD. Given that VStore simulates the complexity of everyday life in an ecologically valid environment, it may be suitable for evaluating functional cognition; however, further research is required to confirm this. VStore has theoretical advantages over other tests in being more engaging than traditional pen-and-paper and computerized batteries; it is fully immersive unlike other similar assessments, such as the VRFCAT, potentially increasing a psychological sensation of being there in a specific (virtual) surrounding [56], and thus enabling the assessment of real time cognitive and behavioral responses to that environment [57]. Furthermore, VStore provides complete experimental control, unlike the MET. Further research is urgently required to confirm age-related findings (ie, predictive validity in early cognitive decline) and establish its reliability and sensitivity to changes in cognition and functional capacity in both healthy and clinical samples.


The authors would like to thank Dr Dan W Joyce for his comments and suggestions on the data analysis. LAP is supported by the UK Medical Research Council (MR/N013700/1) and is a King’s College London member of the Medical Research Council Doctoral Training Partnership in Biomedical Sciences. SSS is supported by the National Institute for Health Research Maudsley Biomedical Research Centre.

Authors' Contributions

LAP contributed to the study design and data collection and analysis and wrote the manuscript. MAM advised on data analysis, manuscript preparation, and review. JP contributed in data collection and read and edited the manuscript. CB contributed in data collection and read and edited the manuscript. JB contributed in VStore software development. TD advised on data analysis and read and edited the manuscript. EM cocontributed to the study design, data analysis, and manuscript preparation and review. SSS cocontributed to the study design, data analysis, and manuscript preparation and review.

Conflicts of Interest

SSS and EM created VStore, with contributions from LAP and technical development from Vitae VR Ltd. King’s College London licensed its rights in VStore to Vitae VR Ltd. Authors LAP, EM, and SSS are entitled to a share of any revenue that King’s College London may receive from commercialization of VStore by Vitae VR Ltd.

Multimedia Appendix 1

VStore courtyard.

PNG File , 218 KB

Multimedia Appendix 2

VStore minimarket.

PNG File , 232 KB

Multimedia Appendix 3

VStore shopping list.

DOCX File , 13 KB

Multimedia Appendix 4

VStore checkout.

PNG File , 208 KB

Multimedia Appendix 5

Equipment specification.

DOCX File , 13 KB

Multimedia Appendix 6

VStore software.

DOCX File , 12 KB

Multimedia Appendix 7

VStore movement parameters.

DOCX File , 12 KB

Multimedia Appendix 8

Cogstate tasks.

PDF File (Adobe PDF File), 447 KB

Multimedia Appendix 9

Technological Familiarity Questionnaire.

DOCX File , 12 KB

Multimedia Appendix 10

Mean and SD for VStore outcomes.

DOCX File , 13 KB

Multimedia Appendix 11

Mean and SD for Cogstate outcomes.

DOCX File , 14 KB

Multimedia Appendix 12

Bivariate correlations between Cogstate and VStore outcomes.

DOCX File , 16 KB

Multimedia Appendix 13

Linear regression model assumptions for VStore outcomes.

DOCX File , 544 KB

Multimedia Appendix 14

Bivariate correlations between the Technological Familiarity Questionnaire and Cogstate.

DOCX File , 13 KB

Multimedia Appendix 15

VStore ridge regression cross-validation fit without the Technological Familiarity Questionnaire.

PNG File , 40 KB

Multimedia Appendix 16

Cogstate ridge regression cross-validation fit without the Technological Familiarity Questionnaire.

PNG File , 43 KB

Multimedia Appendix 17

VStore ridge regression cross-validation fit with the Technological Familiarity Questionnaire.

PNG File , 41 KB

Multimedia Appendix 18

Cogstate ridge regression cross-validation fit with the Technological Familiarity Questionnaire.

PNG File , 41 KB

  1. McKhann GM, Knopman DS, Chertkow H, Hyman BT, Jack CR, Kawas CH, et al. The diagnosis of dementia due to Alzheimer's disease: recommendations from the National Institute on Aging-Alzheimer's Association workgroups on diagnostic guidelines for Alzheimer's disease. Alzheimers Dement 2011 May;7(3):263-269 [FREE Full text] [CrossRef] [Medline]
  2. Palmer BW, Heaton RK, Paulsen JS, Kuck J, Braff D, Harris MJ, et al. Is it possible to be schizophrenic yet neuropsychologically normal? Neuropsychology 1997 Jul;11(3):437-446. [CrossRef] [Medline]
  3. Conradi HJ, Ormel J, de Jonge P. Presence of individual (residual) symptoms during depressive episodes and periods of remission: a 3-year prospective study. Psychol Med 2011 Jun;41(6):1165-1174. [CrossRef] [Medline]
  4. Plassman BL, Langa KM, Fisher GG, Heeringa SG, Weir DR, Ofstedal MB, et al. Prevalence of cognitive impairment without dementia in the United States. Ann Intern Med 2008 Mar 18;148(6):427-434 [FREE Full text] [CrossRef] [Medline]
  5. Tatemichi TK, Desmond DW, Stern Y, Paik M, Sano M, Bagiella E. Cognitive impairment after stroke: frequency, patterns, and relationship to functional abilities. J Neurol Neurosurg Psychiatry 1994 Feb;57(2):202-207 [FREE Full text] [CrossRef] [Medline]
  6. Green MF. Cognitive impairment and functional outcome in schizophrenia and bipolar disorder. J Clin Psychiatry 2006 Oct 15;67(10):e12. [CrossRef]
  7. McIntyre RS, Cha DS, Soczynska JK, Woldeyohannes HO, Gallaugher LA, Kudlow P, et al. Cognitive deficits and functional outcomes in major depressive disorder: determinants, substrates, and treatment interventions. Depress Anxiety 2013 Jun;30(6):515-527. [CrossRef] [Medline]
  8. Bosboom P, Alfonso H, Almeida O. Determining the predictors of change in quality of life self-ratings and carer-ratings for community-dwelling people with Alzheimer disease. Alzheimer Dis Assoc Disord 2013;27(4):363-371. [CrossRef] [Medline]
  9. Liu-Seifert H, Siemers E, Price K, Han B, Selzler KJ, Henley D, Alzheimer’s Disease Neuroimaging Initiative. Cognitive impairment precedes and predicts functional impairment in mild Alzheimer's disease. J Alzheimer's Dis 2015;47(1):205-214 [FREE Full text] [CrossRef] [Medline]
  10. Fusar-Poli P, Deste G, Smieskova R, Barlati S, Yung AR, Howes O, et al. Cognitive functioning in prodromal psychosis: a meta-analysis. Arch Gen Psychiatry 2012 Jun;69(6):562-571. [CrossRef] [Medline]
  11. Groves SJ, Douglas KM, Porter RJ. A systematic review of cognitive predictors of treatment outcome in major depression. Front Psychiatry 2018 Aug 28;9:382 [FREE Full text] [CrossRef] [Medline]
  12. Keefe R, Buchanan R, Marder S, Schooler NR, Dugar A, Zivkov M, et al. Clinical trials of potential cognitive-enhancing drugs in schizophrenia: what have we learned so far? Schizophr Bull 2013 Mar;39(2):417-435 [FREE Full text] [CrossRef] [Medline]
  13. Salagre E, Solé B, Tomioka Y, Fernandes B, Hidalgo-Mazzei D, Garriga M, et al. Treatment of neurocognitive symptoms in unipolar depression: a systematic review and future perspectives. J Affect Disord 2017 Oct 15;221:205-221. [CrossRef] [Medline]
  14. Sevigny J, Chiao P, Bussière T, Weinreb PH, Williams L, Maier M, et al. The antibody aducanumab reduces Aβ plaques in Alzheimer's disease. Nature 2016 Sep 01;537(7618):50-56. [CrossRef] [Medline]
  15. Wolf T, Edwards D, Giles G. Functional Cognition and Occupational Therapy: A Practical Approach to Treating Individuals With Cognitive Loss. Bethesda: AOTA Press; 2019.
  16. Buchanan RW, Davis M, Goff D, Green MF, Keefe RS, Leon AC, et al. A summary of the FDA-NIMH-MATRICS workshop on clinical trial design for neurocognitive drugs for schizophrenia. Schizophr Bull 2005 Jan;31(1):5-19. [CrossRef] [Medline]
  17. Sabbagh MN, Hendrix S, Harrison JE. FDA position statement "early Alzheimer's disease: developing drugs for treatment, guidance for industry". Alzheimer’s Dement Transl Res Clin Interv 2019;5(1):13-19 [FREE Full text] [CrossRef] [Medline]
  18. Bowie CR, Leung WW, Reichenberg A, McClure MM, Patterson TL, Heaton RK, et al. Predicting schizophrenia patients' real-world behavior with specific neuropsychological and functional capacity measures. Biol Psychiatry 2008 Mar 01;63(5):505-511 [FREE Full text] [CrossRef] [Medline]
  19. Millan MJ, Agid Y, Brüne M, Bullmore ET, Carter CS, Clayton NS, et al. Cognitive dysfunction in psychiatric disorders: characteristics, causes and the quest for improved therapy. Nat Rev Drug Discov 2012 Feb 01;11(2):141-168. [CrossRef] [Medline]
  20. Fervaha G, Zakzanis KK, Foussias G, Graff-Guerrero A, Agid O, Remington G. Motivational deficits and cognitive test performance in schizophrenia. JAMA Psychiatry 2014 Sep;71(9):1058-1065. [CrossRef] [Medline]
  21. Rosen WG, Mohs RC, Davis KL. A new rating scale for Alzheimer's disease. Am J Psychiatry 1984 Nov;141(11):1356-1364. [CrossRef] [Medline]
  22. Nuechterlein KH, Green MF, Kern RS, Baade LE, Barch DM, Cohen JD, et al. The MATRICS Consensus Cognitive Battery, part 1: test selection, reliability, and validity. Am J Psychiatry 2008 Feb;165(2):203-213. [CrossRef] [Medline]
  23. Kern RS, Nuechterlein KH, Green MF, Baade LE, Fenton WS, Gold JM, et al. The MATRICS Consensus Cognitive Battery, part 2: co-norming and standardization. Am J Psychiatry 2008 Feb;165(2):214-220. [CrossRef] [Medline]
  24. Morris JC. Clinical dementia rating: a reliable and valid diagnostic and staging measure for dementia of the Alzheimer type. Int. Psychogeriatr 2005 Jan 10;9(S1):173-176. [CrossRef]
  25. Patterson TL, Goldman S, McKibbin CL, Hughs T, Jeste DV. UCSD Performance-Based Skills Assessment: development of a new measure of everyday functioning for severely mentally ill adults. Schizophr Bull 2001;27(2):235-245. [CrossRef] [Medline]
  26. Chaytor N, Schmitter-Edgecombe M. The ecological validity of neuropsychological tests: a review of the literature on everyday cognitive skills. Neuropsychol Rev 2003 Dec;13(4):181-197. [CrossRef] [Medline]
  27. Tarnanas I, Schlee W, Tsolaki M, Müri R, Mosimann U, Nef T. Ecological validity of virtual reality daily living activities screening for early dementia: longitudinal study. JMIR Serious Games 2013 Aug 06;1(1):e1 [FREE Full text] [CrossRef] [Medline]
  28. Shallice T, Burgess PW. Deficits in strategy application following frontal lobe damage in man. Brain 1991 Apr;114 ( Pt 2):727-741. [CrossRef] [Medline]
  29. Keefe RS, Davis VG, Atkins AS, Vaughan A, Patterson T, Narasimhan M, et al. Validation of a computerized test of functional capacity. Schizophr Res 2016 Aug;175(1-3):90-96 [FREE Full text] [CrossRef] [Medline]
  30. Ruse SA, Harvey PD, Davis VG, Atkins AS, Fox KH, Keefe RS. Virtual reality functional capacity assessment in schizophrenia: preliminary data regarding feasibility and correlations with cognitive and functional capacity performance. Schizophr Res Cogn 2014 Mar;1(1):e21-e26 [FREE Full text] [CrossRef] [Medline]
  31. Neguț A, Matu S, Sava FA, David D. Virtual reality measures in neuropsychological assessment: a meta-analytic review. Clin Neuropsychol 2016 Feb;30(2):165-184. [CrossRef] [Medline]
  32. Bird CM, Burgess N. The hippocampus and memory: insights from spatial processing. Nat Rev Neurosci 2008 Mar;9(3):182-194. [CrossRef] [Medline]
  33. Moodley K, Minati L, Contarino V, Prioni S, Wood R, Cooper R, et al. Diagnostic differentiation of mild cognitive impairment due to Alzheimer's disease using a hippocampus-dependent test of spatial memory. Hippocampus 2015 Aug 26;25(8):939-951. [CrossRef] [Medline]
  34. Sheline YI, Wang PW, Gado MH, Csernansky JG, Vannier MW. Hippocampal atrophy in recurrent major depression. Proc Natl Acad Sci USA 1996 Apr 30;93(9):3908-3913 [FREE Full text] [CrossRef] [Medline]
  35. Heckers S. Neuroimaging studies of the hippocampus in schizophrenia. Hippocampus 2001;11(5):520-528. [CrossRef] [Medline]
  36. Vitae VR homepage. Vitae VR.   URL: [accessed 2021-12-11]
  37. Zygouris S, Tsolaki M. Computerized cognitive testing for older adults: a review. Am J Alzheimers Dis Other Demen 2015 Feb 13;30(1):13-28 [FREE Full text] [CrossRef] [Medline]
  38. Wechsler Abbreviated Scale of Intelligence. The Psychological Corporation. 1999.   URL: [accessed 2021-12-11]
  39. Venables W, Ripley B. Modern Applied Statistics with S. Cham: Springer; 2002.
  40. Friedman J, Hastie T, Tibshirani R. Regularization paths for generalized linear models via coordinate descent. J Stat Softw 2010;33(1):1-22 [FREE Full text] [Medline]
  41. Neguţ A, Matu S, Sava FA, David D. Task difficulty of virtual reality-based assessment tools compared to classical paper-and-pencil or computerized measures: a meta-analytic approach. Comput Human Behav 2016 Jan;54(C):414-424. [CrossRef]
  42. Egerházi A, Berecz R, Bartók E, Degrell I. Automated Neuropsychological Test Battery (CANTAB) in mild cognitive impairment and in Alzheimer's disease. Prog Neuropsychopharmacol Biol Psychiatry 2007 Apr 13;31(3):746-751. [CrossRef] [Medline]
  43. Fowler KS, Saling MM, Conway EL, Semple JM, Louis WJ. Paired associate performance in the early detection of DAT. J Int Neuropsychol Soc 2002 Jan 11;8(1):58-71. [CrossRef]
  44. Kéri S, Szamosi A, Benedek G, Kelemen O. How does the hippocampal formation mediate memory for stimuli processed by the magnocellular and parvocellular visual pathways? Evidence from the comparison of schizophrenia and amnestic mild cognitive impairment (aMCI). Neuropsychologia 2012 Dec;50(14):3193-3199. [CrossRef] [Medline]
  45. Ruse SA, Davis VG, Atkins AS, Krishnan KR, Fox KH, Harvey PD, et al. Development of a virtual reality assessment of everyday living skills. J Vis Exp 2014 Apr 23(86) [FREE Full text] [CrossRef] [Medline]
  46. Baddeley AD, Bressi S, Della Sala S, Logie R, Spinnler H. The decline of working memory in Alzheimer's disease. A longitudinal study. Brain 1991 Dec;114 ( Pt 6):2521-2542. [CrossRef] [Medline]
  47. Forbes NF, Carrick LA, McIntosh AM, Lawrie SM. Working memory in schizophrenia: a meta-analysis. Psychol Med 2009 Jun;39(6):889-905. [CrossRef] [Medline]
  48. Chai WJ, Abd Hamid AI, Abdullah JM. Working memory from the psychological and neurosciences perspectives: a review. Front Psychol 2018 Mar 27;9:401 [FREE Full text] [CrossRef] [Medline]
  49. Olson KE, O'Brien MA, Rogers WA, Charness N. Diffusion of technology: frequency of use for younger and older adults. Ageing Int 2011 Mar 28;36(1):123-145 [FREE Full text] [CrossRef] [Medline]
  50. Oliveira CR, Lopes Filho BJ, Esteves CS, Rossi T, Nunes DS, Lima MM, et al. Neuropsychological assessment of older adults with virtual reality: association of age, schooling, and general cognitive status. Front Psychol 2018;9:1085 [FREE Full text] [CrossRef] [Medline]
  51. Yamamoto N, Degirolamo GJ. Differential effects of aging on spatial learning through exploratory navigation and map reading. Front Aging Neurosci 2012 Jun 12;4:14 [FREE Full text] [CrossRef] [Medline]
  52. Hafting T, Fyhn M, Molden S, Moser M, Moser EI. Microstructure of a spatial map in the entorhinal cortex. Nature 2005 Aug 11;436(7052):801-806. [CrossRef] [Medline]
  53. Axelrod BN. Validity of the Wechsler abbreviated scale of intelligence and other very short forms of estimating intellectual functioning. Assessment 2002 Mar 26;9(1):17-23. [CrossRef] [Medline]
  54. Derksen S, Keselman H. Backward, forward and stepwise automated subset selection algorithms: frequency of obtaining authentic and noise variables. Br J Math Stat Psychol 1992;45(2):265-282. [CrossRef]
  55. Broadbent D, Cooper P, FitzGerald P, Parkes K. The Cognitive Failures Questionnaire (CFQ) and its correlates. Br J Clin Psychol 1982 Feb;21(1):1-16. [CrossRef] [Medline]
  56. Slater M. Presence and emotions. Cyberpsychol Behav 2004 Feb;7(1):121; author reply 123. [CrossRef] [Medline]
  57. Eichenberg C, Wolters C. Virtual realities in the treatment of mental disorders: a review of the current state of research. Virtual Reality Psychol Med Pedagogical App 2012. [CrossRef]

AD: Alzheimer disease
AIC: Akaike Information Criterion
CPAL: Continuous Paired Associate Learning
DET: Detection
DV: dependent variable
GMLT: Groton Maze Learning Task
IDN: Identification
ISLT: International Shopping List Task
IV: independent variable
MET: Multiple Errands Test
MSE: mean squared error
OCL: One Card Learning
ONB: One-Back
TFQ: Technological Familiarity Questionnaire
TWO: Two-Back
VR: virtual reality
VRFCAT: Virtual Reality Functional Capacity Assessment Tool

Edited by R Kukafka, G Eysenbach; submitted 01.02.21; peer-reviewed by H Huygelier, R Lundin; comments to author 10.05.21; revised version received 05.07.21; accepted 05.10.21; published 26.01.22


©Lilla Alexandra Porffy, Mitul A Mehta, Joel Patchitt, Celia Boussebaa, Jack Brett, Teresa D’Oliveira, Elias Mouchlianitis, Sukhi S Shergill. Originally published in the Journal of Medical Internet Research (, 26.01.2022.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.