Alzheimer disease (AD) is a neurodegenerative disease associated with gradual decline in cognition, where disease stages range from preclinical AD, in which cognitive decline is absent or only subtle, to mild cognitive impairment (MCI) and dementia [1,2]. Biologically, the disease is defined by pathological changes in amyloid accumulation and neurofibrillary tau protein tangles that appear years before the onset of cognitive symptoms [1-3]. However, it has been widely shown that, even in the preclinical stage, cognition does decline subtly [4-7]. With the increasing number of older adults, it is expected that AD will become a major societal problem, estimated to affect 131 million people in 2050 worldwide [8,9]. The emergence of AD biomarkers allows for the recognition of the disease in the preclinical stage, which has opened a window of opportunity for treatment studies, as interventions are likely to be most beneficial in early stages of the disease. Accordingly, disease-modifying treatments and nonpharmaceutical prevention trials have increasingly focused on preclinical stages of AD over the past years [10-12]. This shift toward the earliest AD stages highlights the need for new tools that provide outcome measures of cognition that are adequate for use in these early disease stages in the realm of early detection, diagnosis, disease monitoring, and evaluation of treatment effects [13-17]. In addition, to enable large-scale decentralized prevention, intervention, or disease-monitoring initiatives, tools are required that enable remote, time-efficient, and reliable assessment of cognition.

The current gold standard to assess cognition in AD is through neuropsychological testing, using paper-and-pencil tests that need to be supervised by a trained neuropsychologist. Importantly, such traditional cognitive tests are not suitable for the earliest AD stages as these tests often lack the sensitivity that is required to detect subtle cognitive changes [7,17,18]. Thus, novel, stage-specific testing paradigms are needed, and digital tools are promising to fill this gap [19,20]. A major advantage of digital cognitive tools is their suitability for unsupervised remote assessment, which enables highly frequent testing that may provide a more accurate reflection of cognition [21,22]. Given the intuitive person-device interaction of touch screen devices, smartphone- and tablet-based tools provide the optimal modality for unsupervised remote assessment [23,24]. Other advantages of such cognitive tools include reduced patient burden; time efficiency through automatic administration and scoring; high scalability; and the potential for rich data collection of precise measurements, including, for example, response times.

Over the last decade, numerous digital tools have been developed that have the potential for remote assessment of cognition in early AD stages [19]. However, the development of such tools has not yet led to large-scale implementation [25]. To date, the measurement quality of digital tools to assess cognition is largely unknown, although it has widely been acknowledged that information on psychometric properties is important for the use and implementation of these tools [19,20,26-28]. For the evaluation of cognitive tests in early AD stages, a recommendation framework has been proposed based on the Consensus-Based Standards for the Selection of Health Measurement Instruments methodology [16,29,30]. This framework highlights the importance of psychometric properties concerning structural and construct validity, responsiveness, and clinical meaningfulness. Hence, available information on psychometric properties is a prerequisite for the successful implementation of digital tools. Although previous reviews have described some validation aspects of digital cognitive assessments [19,31], a comprehensive overview of the reporting of crucial psychometric properties (ie, structural and construct validity, reliability, responsiveness, and clinical meaningfulness) of digital cognitive tools for use in preclinical AD or MCI is currently missing.

This scoping review was conducted to identify the reporting on quality characteristics of smartphone- and tablet-based tools to assess cognition in individuals with preclinical AD or MCI. Our scope was limited to smartphone- and tablet-based cognitive tools with the potential for remote assessment. The primary focus was on the availability of information on psychometric properties, and the secondary focus on the reporting on practical tool characteristics. The findings of this review will be used to formulate recommendations about future steps that are needed to facilitate the implementation of remote smartphone- and tablet-based tools to assess cognition.

This scoping review is conducted in line with a methodological framework for conducting scoping studies [32]. The reporting complies with the guidelines of the PRISMA-ScR (Preferred Reporting Items for Systematic Reviews and Meta-Analyses extension for Scoping Reviews) checklist [33], as shown in Multimedia Appendix 1. The review was not registered, and no protocol was prepared.

After several scoping searches, 4 bibliographic databases (PubMed, Embase [Elsevier], Web of Science Core Collection [Clarivate Analytics], and APA PsycINFO [EBSCO]) were searched for relevant literature from January 1, 2008 (ie, the year that the Google Play Store and Apple App Store, two of the biggest mobile app distribution platforms, were launched), to January 5, 2023. Searches were devised in collaboration with a medical information specialist (KAZ). Search terms including synonyms, closely related words, and keywords were used as free-text words (eg, “Alzheimer,” “digital,” and “cognition”). The search contained no methodological search filter that would limit results to specific study designs or languages. Duplicate studies were excluded using the R package ASYSD (R Foundation for Statistical Computing), an automated deduplication tool [34], which was followed by manual deduplication in EndNote (version X20.0.3; Clarivate Analytics). The full search strategy used for each database is detailed in Multimedia Appendix 2.

Published journal articles were included or excluded according to the criteria outlined in Textbox 1.

To minimize selection bias, 2 of the review authors (RLvdB and SMvdL) independently screened titles and abstracts based on the eligibility criteria (Textbox 1) using ASReview (version 1.1) [35] with default settings (naive Bayes, term frequency–inverse document frequency, and query strategy: max). Differences in judgment were resolved via a consensus procedure through discussion and, if necessary, by consulting a third reviewer (MJK). ASReview is an open-source screening tool using manual input in combination with an active learning algorithm to screen studies based on titles and abstracts. The algorithm was trained using 7 relevant and 7 irrelevant studies, where relevant studies were manually selected and irrelevant studies were randomly provided by the model and subsequently manually labeled as irrelevant. A data-driven stopping rule strategy was followed, which has been suggested previously, and proposes to end the screening process after n consecutive irrelevant records are provided by the active learning algorithm [36]. No consensus exists yet on the optimal n, and the number varies across studies (eg, 20-500) [36-41]. Importantly, it has previously been noted that the stopping rule should aim to find the trade-off between the costs of manually labeling additional records and the costs of erroneous exclusions of relevant records by the algorithm [42]. We stopped the screening process after 100 consecutive irrelevant studies were provided by the ASReview algorithm. This number was based on an initial pilot screening round where it was observed that, if the stopping rule was set to 50 consecutive irrelevant records, relevant records continued to appear after 45 to 49 records, so the cutoff was increased to 100 to include a higher certainty margin. After screening of titles and abstracts, 4 of the review authors (RLvdB, SMvdL, MJK, and AMvG) conducted full-text screening according to the eligibility criteria (Textbox 1) using the web application Rayyan (Qatar Computing Research Institute) [43]. Each study was screened by one of the reviewers independently. Uncertainties were discussed, and if this resulted in disagreement, this was resolved through discussion, or if necessary, a fifth reviewer was consulted.

The systematic literature search generated a total of 23,404 references: 5726 (24.47%) in PubMed, 8069 (34.48%; n=11,322, 48.38% with conference abstracts) in Embase, 7675 (32.79%) in Web of Science Core Collection, and 1934 (8.26%) in APA PsycINFO. After removing duplicates, of the 23,404 references, 11,300 (48.28%) remained. These references were screened based on title and abstract in ASReview. Of the 11,300 abstracts, the first reviewer (RLvdB) manually included 233 (2.06%) relevant ones and excluded 1465 (12.96%) irrelevant ones, and the second reviewer (SMvdL) manually included 234 (2.07%) relevant ones and excluded 1062 (9.4%) irrelevant ones. Within these 2 sets, there was agreement for 33.4% (156/467) of the included abstracts and 38.3% (969/2527) of the excluded abstracts, whereas 1.27% (144/11,300) of the abstracts were labeled differently (eg, included or excluded) such that, within the set of studies that were manually labeled by both reviewers, there was 88% agreement. After reaching consensus, 233 relevant studies were identified and included for further full-text screening. After full-text screening, of the 233 studies, 50 (21.5%) were included for data extraction. The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) flowchart [45] of the search and selection process is presented in Figure 1.

In this scoping review, we identified 50 studies covering 37 different smartphone- and tablet-based tools to assess cognition in individuals with preclinical AD or MCI. These numbers indicate that multiple smartphone- and tablet-based cognitive tools are currently being developed, whereas only a limited number of validation studies in this target population have been conducted for individual tools. Specifically, except for construct validity, most psychometric properties (ie, structural validity, reliability, responsiveness, and clinical meaningfulness) have not been extensively assessed to date. Thus, considering the foundations of measurement, as reflected in the psychometric properties, these tools are still in the early stages of validation. Moreover, it was observed that practical characteristics such as administration time or the investigative setting were not consistently reported. Although the initial reporting on the psychometric properties of smartphone- and tablet-based tools to assess cognition in early AD support their potential, further and careful validation of their psychometric properties, as well as clear information on practical characteristics, is essential to facilitate implementation.

Practical tool characteristics may be argued to be important toward considering the tool’s use in clinical practice, supported by previous observations that, for health care professionals, time efficiency would facilitate the use of digital tools [26]. We highlight that, for a significant number of tools, information was not reported regarding administration time (14/37, 38%), the testing design (single or repeated; 9/37, 24%), or the investigative setting (on-site or remote; 7/37, 19%). This is concerning as information on the setting is crucial to interpret whether the psychometric properties assessed are applicable for on-site or remote use of a tool and for a single (eg, annual) or repeated (eg, daily or monthly) measurement. Especially given that the advantages of digital tools include their time-efficient administration and potential for remote and highly frequent administration, we recommend that such characteristics be clearly reported.

The generation of high-dimensional data is inherent to digital tools that assess cognition. While this may be regarded as an advantage, we observed that it also poses challenges to the overall evaluation of the quality of a tool and complicates one-to-one comparisons between tools. For instance, it remains challenging to decide which scores should be evaluated and compared, which raises questions such as whether the psychometric properties of single scores or aggregated scores should be considered when appraising the quality of the digital tool and whether the tool’s quality is determined by the outcome score that holds the lowest or highest psychometric quality. These challenges of high-dimensional datasets were specifically observed in the psychometric properties of construct validity and test-retest reliability, which widely ranged for the number of generated scores of a single tool. We recommend that the promising large amounts of generated data be reduced carefully by selecting those outcome scores that hold high psychometric quality, which we consider a crucial step to develop digital cognitive tools that provide clinically useful outcome scores. Another observation was that different methodologies were used between tools to determine their psychometric properties. This highlights the need for more uniformity regarding psychometric evaluation, which would facilitate the comparison of digital tools regarding their psychometric quality, which is needed to enable the end user to select the most appropriate tool for its context of use.

To standardize the process of clinical evaluation of digital tools, we propose a more widespread use of operating procedures, such as the FDA guidance on software as a medical device [95]. This guideline recognizes the importance of psychometric property evaluation and recommends 3 steps. The first step concerns clinical association, where the digital outcome should be demonstrated to be associated with the targeted clinical condition (ie, clinical decline as a result of AD). The second evaluation step regards analytical validation of the software, which is out of scope for this review. The third step involves clinical validation, where the digital outcome should be demonstrated to be reliable and clinically meaningful with regard to its intended purpose. The importance of provided information on psychometric properties is further emphasized by the previously reported notion that clear information on psychometric properties would facilitate the use of digital tools in clinical practice [26]. To this end, we will discuss the reporting on individual psychometric properties in the following paragraphs.

A first psychometric property that could be argued to be placed at the first step of the clinical evaluation process regards the conduct of factor analyses to confirm that outcome scores adequately reflect the dimensionality that is measured (ie, the underlying structure of one or multiple cognitive domains). In this review, we observed that a factor analysis on outcome scores was conducted for a limited number of digital tools (3/37, 8%) to demonstrate the underlying structure of the measured construct. This is concerning, and it is advisable for future validation studies to increasingly focus on conducting factor analyses to ensure the digital tool’s structural validity.

Another psychometric property regarding demonstrating clinical associations is construct validity, which was reported for most digital tools (36/37, 97%). Correlation levels between generated digital scores and scores on gold-standard measures of cognition ranged significantly both within and between tools, similarly to previous findings regarding self-administered computerized cognitive assessments [31]. It has been highlighted that, as currently used cognitive tests may not be suitable for early AD stages, validation against these “gold standard” tests may be particularly relevant in more progressed stages, whereas validation against biological measures of AD may be more relevant in preclinical stages [16,17,20]. Recently, proposed revisions to the diagnostic criteria for AD have been published, with the common denominator that preclinical AD concerns a stage characterized by the absence of symptoms [96,97]. Hence, to demonstrate an association with the disease in this stage, biological validation is needed instead of clinical validation, for which no gold-standard test exists at this stage. This importance of biological validation in the preclinical AD stage was reflected in our observation that validation against biological measures was reported for fewer tools (10/37, 27%) and all tools that were assessed in the preclinical stage were validated against biological measures. Still, most tools (36/37, 97%) were validated against clinical measures, including tools that were examined in preclinical AD. In the absence of a true gold standard for preclinical AD, it remains concerning that novel sensitive tests for preclinical populations are nonetheless validated against existing tests with limited sensitivity. As suggested previously, predictive validity might be considered as an alternative “copper standard” [16]. As we did not evaluate the reporting on predictive validity, further research should examine the ability of smartphone- and tablet-based tools for use in early AD stages to predict future cognitive decline or progression to MCI or dementia.

The third clinical evaluation step, clinical validation, involves demonstrating reliability. Clinical trials examining potential intervention effects inherently require repeated assessments over time, and to accurately interpret change in cognition over time, the test-retest reliability of an outcome measure should be ensured in the target population. The test-retest reliability was examined for a limited number of digital tools included in this review (12/37, 32%). As we selected studies that focused on the MCI and preclinical stages, it may be the case that test-retest reliability was assessed in studies focusing on other populations, including, for example, individuals with dementia. However, as test-retest reliability may differ across populations, it is important that this psychometric property is assessed in the target population. The importance of ensuring the tool’s reliability may be emphasized even more in the context of self-administered digital tools that may be more susceptible to noise than supervised paper-and-pencil tests. For the tools that had available information on the test-retest reliability, the reliability levels ranged substantially, largely depending on the outcome score of a tool (eg, total score or single scores), in line with previous observations [31]. This makes it highly challenging to determine the overall reliability of a specific tool and stresses the importance of careful selection of outcome scores. We recommend that the test-retest reliability should be thoroughly considered before selecting outcome scores for further validation steps. In addition, as the large absence of information on this psychometric property may hinder the implementation of digital tools, validation studies should increasingly focus on evaluating test-retest reliability. Alternatively, although we note that the independent validation studies are preferable, we recommend that the evaluation of test-retest reliability also be incorporated into clinical trial designs where, for example, the natural course of a digital outcome should be attested first before investigating its change after the intervention.

To determine whether a digital cognitive tool is sensitive to measure potential treatment effects or cognitive changes over time for monitoring purposes, information on the tool’s responsiveness is essential. Responsiveness was reported for a small number of tools (6/37, 16%) using different methodologies, and findings varied within the outcome scores of a single tool. Interestingly, of these 6 tools, 2 (33%) showed that less steep improvement was associated with preclinical AD [78,81]. Thus, although it may be expected that declining performance would be indicative of cognitive change, these new digital repeated testing approaches showed that diminished learning curves may also serve as an indicator of cognitive change in individuals with preclinical AD. Our finding that responsiveness was evaluated to a lesser extent may be explained by the fact that it requires a more effortful, longitudinal study design. In addition, this low number of tools for which responsiveness was reported may reflect the relatively young character of the research field involving digital tools to assess cognition in early AD stages, which may indicate that these tools are not ready for clinical implementation yet and warrant further validation. Accordingly, it is recommended that future validation studies focus on examining the responsiveness of digital outcome scores to change over time in the target population as an essential step toward clinical implementation.

Another aspect of the third clinical validation step regards clinical meaningfulness. To enable clinical interpretation of digital scores and clinically relevant change, the clinical meaningfulness of scores and their change should be determined. However, no consensus has been established yet on what score should be considered clinically meaningful [98]. The importance of clinically meaningful end points has also been stated by the US Food and Drug Administration draft guidance from March 2024 [99], where it has been recognized that, in early AD stages, it may be challenging to demonstrate clinical meaningfulness due to subtle cognitive changes and the absence of overt cognitive impairment. With the lack of a clear definition of clinical meaningfulness in early AD stages, this quality characteristic may be easily overlooked in the validation of digital cognitive tools, as reflected in our observation that clinically meaningful scores were not defined for any of the identified digital tools. However, it should be noted that, for some digital outcome scores, it was argued that their meaningfulness was supported by associations with AD biomarkers and clinical measures (PACC or Clinical Dementia Rating), as well as by scores or learning curves that differed between diagnostic groups [71,78,80]. Such definitions of clinical meaningfulness related to clinical, biological, and discriminative validity seem to refer to the relevance of the measure rather than the magnitude of the score or change in score that demonstrated clinical meaningfulness [100]. Hence, there is a pressing need for a clear definition of clinical meaningfulness, and it has been suggested that this definition should be developed by incorporating perspectives of various stakeholders (eg, patients, clinicians, and regulatory bodies) [98]. In addition, clinical meaningfulness may differ across patients, for which digital tools may offer an opportunity to determine personalized end points. Moreover, norm scores or cutoff scores may be considered toward establishing clinical meaningfulness. Thus, it is recommended that, within the field concerning cognitive assessment in early-stage AD in general, attention should be paid to reaching consensus on the definition of clinical meaningfulness of digital outcome scores to enable clinically useful interpretation of digital outcomes in future validation studies.

The major strength of this review was that we focused on the preclinical AD and MCI stages. These predementia stages offer the greatest potential for benefit from intervention strategies as well as having the highest need and offering the greatest feasibility in the context of novel digital remote assessments. Another strength was that we used a systematic search to identify validation studies of digital tools to assess cognition, which allowed for a large selection of studies. In addition, as our scope covered a range of psychometric properties that were selected based on previous recommendations, as well as practical tool characteristics that enabled the identification of an “information gap” in both areas, this allowed for the provision of a clear overview on what information is missing and, thus, what should be focused on to support the implementation of digital tools.

This review has a number of limitations. First, although we used a broad and systematic search strategy to minimize the chance of missing studies, the identified studies relied on specific search terms such that potentially relevant studies might have been excluded. For example, studies were not identified within the search if keywords related to the digital aspect of tools, such as “digital,” “computerized,” “smartphone,” or “tablet,” were not used in the title or abstract. Second, we did not conduct quality assessments of the included studies, which may have potentially biased our overall conclusions. Third, because this is a rapidly developing field, new digital cognitive assessments will be continuously emerging, highlighting the need for ongoing evaluation. Therefore, our aim was not to provide an elaborate overview of currently available tools but rather identify possible information gaps on reported psychometric properties in the landscape of digital cognitive tools. In addition, our scope was limited to tools that had the potential for remote assessment. The evaluation of this characteristic was not based on a strict criterion but rather on the contextual information provided indicating the suitability for self-administration that implies the potential for remote administration, which may, thus, be slightly subjective. We decided not to formulate a stricter eligibility criterion for this remote design as, for most tools, it was not explicitly stated whether the tool was intended for remote assessment. As this is in fact important information, it is recommended that future validation studies provide clear information on the intended context of the digital tool. Moreover, as we focused on validation studies, we did not include development studies, nor did we assess additional information on the tools’ websites on the tool characteristics, such as administration time, intended use, or available languages. Therefore, we emphasize that the provided overview of such characteristics may not adequately reflect all practical information or languages available to date but rather provide an overview of all available information as reported in validation studies targeting a population of individuals with preclinical AD or MCI. However, it is crucial to include such information in scientific publications for end users to decide which tools are most appropriate for their context of use and to evaluate the quality of digital cognitive tools in future systematic reviews [101]. Furthermore, we did not evaluate the quality of the methods used to assess the psychometric properties, and it is subject to discussion whether any method used was appropriate. For instance, to demonstrate structural validity, confirmatory factor analyses are argued to be the preferred method over exploratory factor analyses [102-104]. Similarly, to evaluate the quality of the tool’s construct validity, it is of relevance whether the clinical or biological correlations or group differences assessed are in the hypothesized direction, and for evaluating responsiveness, the time frame (eg, days, weeks, or months) should be considered. We provided a first step to identify the reporting on psychometric properties regardless of their quality, but future research should focus on the quality of the methodology used to determine psychometric properties.

Our results indicate that the landscape of smartphone- and tablet-based cognitive tools for early AD could currently be placed at the first step in the validation process, as described in the Food and Drug Administration guidance on software as a medical device [95]. Following this guidance, it is recommended that, after establishing a valid clinical association, digital tools should be further validated with regard to their context of use. For the development of novel tools, we recommend clearly stating the digital tool’s intended use to determine the prioritized validation steps with regard to the most relevant psychometric properties. For instance, if a tool would be used for monitoring purposes, demonstrating responsiveness is crucial. For digital screening tools, sensitivity should be demonstrated, whereas for digital diagnostic tools, the specificity for a clinical condition should be validated. As we identified multiple gaps in the available information on crucial psychometric properties, which halts the clinical validation of these tools, this stresses the need for more attention on psychometric quality, where the selection of outcome scores should be considered carefully and reported consistently. As a next step, future studies should determine the psychometric quality of currently developed digital tools for use in early AD stages. Ensuring psychometrically high-quality outcome scores must be a central theme in the development of smartphone- and tablet-based tools that assess cognition in individuals with preclinical AD and MCI. Further clinical validation tailored to the context of use will pave the way for the implementation of digital cognitive tools.