All participants were recruited and evaluated at the Tongji Hospital, affiliated with Tongji University, Shanghai, China. Data collection took place between January 2018 and October 2021. Following the comprehensive neuropsychological assessment tests based on 2011 National Institute on Aging-Alzheimer’s Association guidelines [22], brain magnetic resonance imaging or computed tomography scan, and serum blood tests, our team, comprising board-certified psychiatrists and neurologists, categorized participants into 3 groups: healthy controls (HC), participants diagnosed with MCI, and participants diagnosed with AD. The exclusion criteria were as follows: (1) any lifetime history of stroke, head injury, substance abuse, or major or medical psychiatric disorders; (2) large intracranial vessel stenosis >50%; and (3) being unable to cooperate with neuropsychological tests. We conducted a 1-year follow-up to assess their cognitive function, including executive function (TMT), and re-evaluated their cognitive diagnosis and executive function.

Ethics approval (#2021-081) was granted by the institutional review board of the Ethics Committee of Shanghai Tongji Hospital in China and complied with the principles of the Declaration of Helsinki. All participants were informed about the study and the confidentiality of their data and signed a consent form before participating in the study at baseline and follow-up. All data used in this analysis were deidentified. There was no compensation for participating in this study.

The combination of paper and electronic TMTs retains the same psychometric properties as the standard TMT but provides more abundant information for quantitative assessment, such as wrist velocity and pressure level. In this study, participants were given a printout of the TMT on A4 paper and were instructed to draw lines to connect the circles or squares in ascending order. The A4 paper was placed on an electromagnetic tablet (PH-1820-A; PendoTech). This combination approach not only records abundant information but also provides a better interactive experience for older adults who are not familiar or comfortable with electronic presentations. The trajectory points collected by the digital tablet consisted of 4 variables, including Cartesian coordinates (x and y), pressure, and time stamps. The Cartesian coordinates represented the position of the trajectory points on a 2D plane, where x ranged from 0 to 21,000 and y ranged from 0 to 29,700 because of the limited tablet size (210 × 297 mm). The pressure variable indicates the strength of the drawing, and the time stamp is the sampling time of the point. When the pen is lifted in the air, the pressure becomes zero, which is highly beneficial for calculating the preparation time or thinking time. A flowchart of the study is shown in Figure 1.

The TMT was initially developed by Zhao et al [23] in 2013 with the aim of eliminating reliance on the Latin alphabet. In this study, the TMT (also called the Shape Trial Test) assessment had a standardized format and was formulated and administered by well-trained neuropsychologists. In Part A, the participants were instructed to draw a line between circles or squares as rapidly as possible, joining consecutive numbers. Part B displayed all the numbers twice, except for the number 1 (encircled by a square), with each corresponding number encompassed in both circles and squares. Hence, Part B was more demanding than Part A for visual perceptual processing ability because of greater visual interference and longer path length [19]. Parts A and B had 2 blocks: in the first block, the numbers ranged from 1 to 8, and in the second block, the numbers ranged from 1 to 25. In the first block, the participants were allowed to attempt the test without restraint. The short warm-up blocks introduced participants to an understanding of the TMT-square and circle. However, in the second block, several rules were introduced to regulate the participants’ behavior. Similar to the paper-based TMT, if a connection error occurred, the examiner pointed it out and allowed the participant to correct it. When the participant was confused about the next target to connect for more than half a minute, the examiner prompted the participant. In addition, to record complete trajectories, participants were cautioned whenever they lifted the pen from the paper. Finally, the number of connection errors, prompts, and pen-up warnings was recorded.

The entire cohort (N=300) was stratified and randomly sampled into a subset for model development and validation, with 100 participants in each group. Three participants with incomplete data were excluded. The demographic characteristics of the participants enrolled in the study are shown in Table 1. We used chi-square tests and one-way ANOVA to assess the independence of categorical variables between each pair of groups. A highly significant association was observed between the HC and AD groups in terms of age (P<.001) and educational level (P<.001) and between the HC and MCI groups for age (P=.24). In addition, there were no significant between-group differences for sex (P=.06).

To investigate the associations between each feature and cognitive decline associated with cognitive impairment, a one-way ANOVA was used to explore whether differences existed between the groups. As shown in Table 2, time-related and writing pressure–related features were significantly associated with cognitive ability (P<.001), whereas a lower significance for the jerk feature indicated a weak discriminating ability (TMT-A-1, HC vs AD, P=.57). Meanwhile, for TMT-B-2 in particular, the higher significance indicated that the TMT-B-2 had a more powerful ability to identify cognitive impairment in patients (P<.001).

We also explored the development of a supervised machine learning model to validate the efficiency of these features. A 5-fold cross-validation was performed to evaluate the performance and applicability of the decision-making model. First, we implemented and compared different candidate algorithms to screen for a robust learning model, including support vector classification, adaptive boosting, random forest, gradient boosting decision tree, and light gradient boosting machine. The hyperparameters of the classifiers were maintained at default values. In the future, a well-trained model will eventually become a tool for ascertaining patients’ impaired cognition in the clinical environment. As shown in Table 3, the random forest model had the highest accuracy and area under the curve among the popular machine learning algorithms. Hence, the random forest model was used as the final predictive model for further analysis.

Then, we characterized the benefit of mixing paper and electronic TMTs via a quantitative comparison of the performance of conventional features and all the abovementioned features. The former is commonly used in the traditional paper-based TMT, which involves completion time, demographic features (sex, age, and educational level), and the number of interactive behaviors (the number of mistakes, pen-up warnings, and prompts). It is worth noting that the number of interactive behaviors was not incorporated into the proposed feature space because the proposed paper-and-electronic TMT was expected to eliminate dependency on caregivers. A performance comparison is presented in Table 4. This result demonstrates that the proposed feature extraction method is highly beneficial for improving diagnostic ability.

To intuitively highlight the contribution of each feature to the prediction of the cognitive impairment level, we used the Shapley additive explanations (SHAP) method [26] (implemented by the Python package SHAP, version 0.41.0) to visualize the variable importance. Specifically, we computed the mean absolute SHAP value for each variable as the importance index. As depicted in Table 5, the proposed features play an important role in cognitive impairment screening, particularly time-related features, writing pressure–related features, and VQ without relative weighting. Meanwhile, the TMT-B provided a better measure of cognitive flexibility and produced more discriminative features than the TMT-A, which is consistent with previous studies.

In addition, to verify the stability and robustness of the proposed method, participants were encouraged to complete the follow-up within 1 year of the first test. We trained the random forest classifier using the initial data collection and validated it during the follow-up assessments (HC group: n=38, MCI group: n=32, and AD group: n=22). Notably, features containing identity-related information, such as sex, age, and educational level, were excluded to avoid data leakage. As listed in Table 6, the well-trained classifier achieved high stability and accuracy in distinguishing the AD group from the HC and MCI groups.

By analyzing 389 participants, we developed and validated a novel combination mode of paper and electronic TMTs, which not only maintained the traditional interaction style for participants who were not familiar or comfortable with electronic presentations but also offered both reproducible and abundant information for automated cognitive assessment. Furthermore, we obtained excellent classification accuracies of 0.726 (HC vs MCI), 0.929 (HC vs AD), and 0.815 (MCI vs AD) using the random forest model. In addition, we also validated our model with the participant’s follow-up data, and we obtained accuracies of 0.656 (HC vs MCI), 0.891 (HC vs AD), 0.857 (MCI vs AD), and 0.929 (AD vs others). We also suggested 4 types of features that were associated with cognitive decline. The experimental results demonstrate the effectiveness of the model in enhancing the accuracy of cognitive impairment screening.

Regarding the correlations between the extracted features and cognitive decline, we used ANOVA to investigate the significance of these features. Both conventional demographic characteristics and time-related features were significantly correlated with cognitive impairment. In addition, the digitization of the TMT allowed us to leverage the recorded data from a different perspective, such as writing pressure and jerk. Writing pressure–related features were significantly correlated with cognitive impairment levels, whereas jerk was slightly correlated. This is in line with a similar previous study [27], which showed that handwriting pressure plays an important role in cognition decline.

The TMT is a widely used neuropsychological test to assess the cognitive function of patients. Sakai et al [28] found that the degree of collapse in the velocity profile shape increased significantly when cognitive function decreased. However, the TMT has limitations: the underlying executive functions articulated during the task are not well discriminated, making it a test with low specificity [29]. Second, in the traditional TMT, only total time is quantified, which does not allow for a detailed analysis. Third, there was a fixed spatial configuration for each condition. We combined the electronic and paper versions of the TMT to overcome these main limitations and evaluated them in a group of older adults with cognitive impairment.

The significance test suggests that the captured features can help improve the performance of automatic screening models, which was also validated by comparing the performance of the machine learning model trained with conventional features and the suggested features. Moreover, the significant differences between the groups indicated that TMT Part B was a more useful tool for measuring the level of cognitive impairment, which is consistent with previous research. A longitudinal study suggested that conversion appears to be less driven by changes in the neurobiological trajectory of the disease than by a sharp decline in functional ability and, to a lesser extent, by declines in executive function [30]. A greater decline in executive function has been shown to be associated with greater ventricular enlargement and volume loss in the frontal, parietal, and temporal lobes [31].

With the development of artificial intelligence and computing technology, the use of digital technology to automatically analyze and assess cognitive function has attracted the attention of researchers because of its objectivity and potential to alleviate the shortage of well-trained physical therapists [5,13,32-34]. Different machine learning algorithms have been used in clinical disease diagnosis, such as deep neural networks [35], logistic regressions, k-nearest neighbors, support vector machines, and naive Bayes classifiers [36]. This study also used 5 common machine learning methods to develop the model. AD is mainly characterized by a dynamic process of neurocognitive changes from normal cognition to MCI and progression to dementia, with the jerk of the TMT also being dynamic. Therefore, in our study, we used a novel trajectory modeling approach based on metric learning (generalized metric learning VQ) [21,37] to extract trajectory features that closely resemble realistic clinical data. This represents a pivotal aspect of our study. The importance of the features also shows that template-based features play an important role in cognitive impairment screening, especially for VQ with relative weighting (one of the top-3 most important features). By incorporating all the features, we observed an Improvement in classification accuracy, suggesting that the electronic TMT features provide more scientifically informative data and hold greater potential for clinical application. We present preliminary evidence suggesting that the proposed combination mode of paper and electronic TMTs is user-friendly, practical, and effective. Our future study plan will focus on the development of realistic applications that hold clinically significant implications for at-home health care.

This study had certain limitations. First, it was a single-center study, and most participants lived in Shanghai. Thus, future research should include multicenter cooperative studies to account for regional and racial differences. Second, our findings were based on the TMT-square and circle, and other types of TMTs remain unexplored. Finally, further research is needed to obtain a clear understanding of how these suggested features relate to the neural changes underlying cognitive impairment.

We propose a novel combination of paper and electronic TMTs, which is expected to not only retain the traditional interaction style for participants who are not familiar or comfortable with electronic presentations but also offer abundant information for automated cognitive assessment. Further, we have proposed 4 types of features associated with cognitive decline for screening. The results demonstrate the effectiveness of this approach and suggest its potential to contribute to the development of a practical tool for assessing cognitive problems in a clinical environment.