The selection of the cognitive tasks included in our battery was based on a literature review of available computerized neuropsychological batteries [17,18]. Three important considerations guided our choices. First, the tasks had to be feasible in the context of an entirely Web-based battery (in relation to connection speed limitations, the use of mouse and keyboard instead of stylus pens, etc). Second, given our objective to study long-term age-related cognitive decline starting from midlife, the tasks had to reflect cognitive processes that have been shown to show slight functional changes in early aging (ie, episodic memory, attention/working memory, and highly integrated executive functions) [16,19]. Third, the selected tasks had to be derived from robust, previously validated neuropsychological paradigms. The tasks that were finally chosen to be part of our battery are described in Table 1. Briefly, the first task (“Click”) consists of connecting numbers (in ascending order) as fast as possible, the second task (“Maze”) consists of discovering a labyrinth route while respecting a certain number of rules, the third task (“Cards”) is a matching-to-sample task presented as a card game, and the fourth task (“Marbles”) consists of memorizing and recalling the color pattern of several marbles with a specific location on the screen. The specific realizations of all tasks except the Click-task are randomly generated at the beginning of each session and were thus not identical across the two modes of administration (fully self-administered/supervised). For example, participants were confronted with distinct labyrinth routes for each of the two completions of the test battery (fully self-administered/supervised).

In order to obtain our final Web-based instrument, the NutriCog battery, the following working steps were undertaken: (1) adaptation of the task instructions to a self-administered computerized framework, (2) prototype development, (3) pilot testing, and (4) revision of the prototype. Pilot testing consisted of assessing comprehensibility of the instructions, timing, and potential technical glitches. These pilot tests have been conducted in our research institute, within a sample of individuals with varying age, education level, and sex.

Lessons that we have learned during the development of the operational version of our battery include the importance of constructive exchanges within an interdisciplinary team of neuropsychologists, epidemiologists, and computer scientists in order to find solutions that are relevant in terms of neuropsychological paradigms, the planned epidemiological investigations, and feasibility in terms of software development. Moreover, multiple rounds of pilot testing within samples of individuals with diversified characteristics were necessary to identify software errors and comprehensibility problems, and to calibrate the display times, for example, messages during the Maze task or cards during the Cards task.

In order to compare the self-administered mode of the NutriCog Web-based cognitive battery to a supervised mode of this same battery, we conducted a comparison study in a subsample of the NutriNet-Santé cohort. Each participant of this subsample was asked to perform the test battery twice (ie, in both the self-administered mode and the supervised mode), with an intermission of about 2 months (mean 73.2, SD 17.2 days) in between the two realizations. In order to account for learning effects that have to be expected when repeatedly administering cognitive tests [16], we randomized participants into two study groups, differing in the order of administration of the test battery version. The respective study groups were named SA-SU (self-administered version first, supervised version second) and SU-SA (supervised version first, self-administered version second). The NutriCog test battery was available on the NutriNet-Santé personal Internet page of each participant. The volunteers were asked to use at least 12-inch monitors for optimal visualization of the tests, to use a mouse, and to be in a quiet place without any disruption.

Prior to administering the battery of cognitive tests, the participants were asked to complete a short questionnaire assessing their current mood. At the end of each round of assessment (ie, self-administered version and supervised version), participants were presented with a process evaluation questionnaire in order to provide feedback on the test battery. The expected time to complete the battery in full was 20-25 minutes (15-20 minutes for the cognitive tests and 5 minutes for the questionnaires). Finally, after the participants had completed both versions of the test battery, they were presented with a satisfaction questionnaire, designed to collect information on the acceptability of the duration of each version of the battery, the difficulty of the tasks, the presentation and comprehensibility of the instructions, and the overall appreciation of the test battery.

The self-administered version was completed by the volunteers while alone, following the instructions given on the webpage. The supervised version was completed with assistance by a trained neuropsychologist, who made a home-visit appointment with each volunteer. The instructions were given orally by the neuropsychologist, who was also available to answer any upcoming questions concerning the instructions. All administrations of the supervised version of the battery were realized by the same neuropsychologist. There were no other differences between the two assessment rounds.

The selection process for our study sample is presented in Figure 1. We selected a subsample of 1416 participants of the NutriNet-Santé cohort, via a stratified randomization procedure, using sex, age group (<50 years, ≥50 years), and educational level (≤2 years of higher education, >2 years of higher education) as stratification variables. Among those subjects who agreed to participate, we selected 208 individuals with varying sex, age, and educational level (our pre-defined objective was to obtain a final sample of approximately 200 participants) and randomly attributed them to the two study groups. Of these 208 subjects, 14 were excluded because they did not validate both versions of the test battery, and 5 individuals were excluded because serious technical (ie, computer-related) problems had occurred as they completed the battery. Thus, our final study sample consisted of 189 participants (group SA-SU: n=99; group SU-SA: n=90).

Participant characteristics were presented as n and % or p50 and p25, p75 (ie, 50^thand 25^th, 75^thpercentile) and differences across study groups were tested via chi-square tests and (nonparametric) Wilcoxon rank-sum tests, respectively. In order to detect potential learning effects, median performances on the first administration and on the second administration of the battery were tabulated and compared via nonparametric paired tests (Wilcoxon signed-rank tests). Furthermore, we tabulated median performances according to test battery version (self-administered, supervised) as p50 and p25, p75. Finally, Spearman correlations between the different cognitive outcomes variables were tabulated.

Concordance between the two versions of the test battery was estimated via Spearman correlation coefficients (presented with their P values). We decided to use Spearman correlations as our main statistical indicators in order to facilitate a comparison of our results with other methodological articles on computerized cognitive test batteries, which mostly report Pearson or Spearman correlations [18]. Spearman correlations were more adapted to our data than Pearson correlations as the cognitive outcome variables were not normally distributed.

Spearman correlations were presented for the whole study sample and stratified by age (<50 years, ≥50 years), educational level (≤2 years of post–high school education, >2 years of post–high school education), sex, and Web experience (Web novice, Web expert). Participants reporting that they were “inexperienced” or “beginners” in terms of Internet knowledge were designated as “Web novices” and participants report “advanced” or “expert” Internet knowledge were designated as “Web experts.”

Finally, participants’ responses to the satisfaction questionnaire were tabulated as n and %. All analyses were conducted with SAS, version 9.3, and all statistical tests were two-sided with a significance level of .05.

The aim of this comparison study was to assess the concordance of the fully self-administered mode of the Web-based NutriCog test battery with a supervised mode of the same battery, characterized by the presence of a trained neuropsychologist. Nine different outcome variables were evaluated to measure the performance of participants on four different tasks (Click, Maze, Cards, and Marbles). Spearman coefficients for the correlation between the two modes of administrations, in terms of cognitive test performances, ranged between .42 and .73. Correlation coefficients were lower for the Maze task (Mazes A and B) than for the other tasks. This could indicate potential problems concerning the comprehensibility of the Maze task instructions in the absence of a supervisor. However, based on suggestions made by the participants and by the neuropsychologist who was present during the supervised version of the test battery, these instructions have been slightly revised (after the comparison study was completed), in order to enhance understanding.

The observed correlation coefficients varied according to educational level and Web knowledge. Correlations tended to be higher among higher educated participants and among those with higher Internet knowledge. However, as can be expected in the context of an entirely Web-based study, there were only very few individuals who reported being “inexperienced” or having a “beginner level” with respect to Web knowledge.

Overall, in the context of this comparison study, very high values of concordance could not be expected due to multiple circumstances. First, we observed a clear learning effect, with test results that were almost systematically better at the second administration of the test battery, independently of the mode of administration. If such learning effects were of similar magnitude for the whole study sample, this would not affect the calculated Spearman correlations. On the other hand, differential learning effects would have lowered the observed concordance values in our sample. Further, cognitive performances are subject to a rather large amount of intra-individual variation [21].

To the best of our knowledge, no other study has yet investigated the concordance of a fully self-administered version of a cognitive test battery with a supervised version of the same test battery. However, the context of our study is similar to studies investigating test-retest reliability (ie, the correlation of performances on a first and a second administration of the same test battery), which have been conducted for multiple computerized cognitive batteries. A systematic review of the literature by Tierney et al identified 11 computerized cognitive test batteries [18], of which nine had information on the test-retest reliability for each subtest. The respective ranges of (Pearson or Spearman) correlation coefficients were .30-.74 (CAMCI) [22], .65-.88 (CANS-MCI) [23], .56-.90 (CNS Vital Signs) [24], .53-.93 (Cognitive Drug Research Computerized Assessment System for Dementia, COGDRAS-D) [25], .23-.79 (Cogstate) [26], .68-.80 (CSI) [27], .59-.98 (short form of the MicroCog battery) [28], and .40-.84 (Mindstreams Mild Impairment Battery) [29]. For the CANTAB battery, intraclass correlation coefficients were reported instead of Pearson or Spearman correlations, with a range of .09-.86 [30]. Of note, in our study, intraclass correlation coefficients (which were calculated after applying transformations to improve normality) ranged from .36-.65 (data not shown).

In this study, lower correlation coefficients than for such test-retest investigations had to be expected as we compared two different modes of administration. Although very high values of ≥.9 for specific subtests (as observed for the CNS Vital Signs, COGDRAS-D, and Microcog batteries) were not obtained in this investigation, the range of correlation coefficients found in our study (.42-.73) is roughly comparable to the ranges found in other studies.

Another element that supports the use of our Web-based cognitive battery in a fully self-administered mode is the fact that a majority (88.7%) of participants evaluated the test instructions as sufficiently detailed and understandable. As stated above, in order to further improve the comprehensibility of the test battery, the instructions concerning the Maze task have been slightly modified by taking into account the suggestions of participants of this comparison study. Besides, 81.1% of participants preferred the supervised version of the battery to the self-administered version. However, this is probably largely due to the fact that social interaction with a health professional was perceived as a more pleasant situation than completing the test battery alone.

A certain number of limitations to this study have to be considered. First, the observed learning effect between the first and second administration of the test battery is difficult to separate from differences in cognitive performances that are related to the mode of administration. Second, this comparison study aimed only to compare the fully self-administered mode of the battery to a supervised mode of administration. Data on the ability of the battery to accurately discriminate normal cognitive function from impaired cognitive function are not available. An important strength of our study is its originality, as to the best of our knowledge, no other study has yet compared full self-administration to supervised administration of cognitive batteries. Moreover, the study was conducted within a rather large sample of 189 participants of varying age, sex, and educational level. Finally, the concordance between the two modes of administrations was assessed for the whole population as well as for specific subgroups.

The concordance of the self-administered version and the supervised version of the Web-based NutriCog cognitive test battery was roughly similar to that observed for test-retest investigations of other test batteries. This indicates that these two different modes of administration provide similar information. In large epidemiological studies like the NutriNet-Santé cohort, the objective of cognitive evaluations is not to provide data with high validity in a clinical context, but to measure cognitive performances in a rapid and simple manner, with sufficient quality to permit valid conclusions on the population level. Given the drastic reduction of the logistical and financial burden that can be obtained by using fully self-administered tools, Web-based cognitive test batteries such as NutriCog provide interesting alternatives to supervised tools in the context of large cohort studies.