The original sample included 9253 observations from 5189 participants who completed at least one NP assessment that was voice recorded. A subset of participants had multiple recordings over the course of the study period. Each digital voice recording and the corresponding NP tests were treated as 1 observation. Exclusion criteria included those observations with missing education information (n=492), prevalent dementia (n=313), flagged as potential MCI but have not gone through dementia review (n=551), and those whose voice recording was less than 10 minutes in length (n=23).

The Institutional Review Board of the Boston University Medical Campus approved the procedures and protocols of the Framingham Heart Study (FHS is H-32132). All participants provided written informed consent.

The details of FHS NP test administration have been reported previously [22]. Multiple cognitive domains are measured by 18 different tests [23-27] including verbal memory, verbal fluency, visual memory, attention and concentration, executive function, abstract reasoning, visuoperceptual organization, and language, as is illustrated in Table 1.

Since 2005, the FHS has been digitally recording all spoken responses during NP test administration, which encompasses the verbal interactions between the tester and the participant. This study included digital voice recordings obtained from September 2005 to March 2020. OpenSMILE software (version 2.1.3) [28] was used to extract an acoustic feature set [29], which contains 65 low-level descriptors (LLDs) from these recordings. This acoustic feature set covers a broad range of information of the voice recordings including pitch, voice quality, loudness, signal energy, waveform, auditory, fast Fourier transform spectrum, spectral, and cepstral, which has been described in detail in a prior study [30]. The feature set has also been used in many fields, such as speech processing, music information retrieval, and emotion recognition [31]. The description of these features is summarized in Table S1 in Multimedia Appendix 1. More details of these features can be found in the previous publication [30]. There are some audio recordings with 1 channel (mono; n=4738), and the others were recorded with 2 channels (stereo; n=3136). For the recordings with 2 channels, we included the first channel in the analysis. Each recording was divided into segments of 20 milliseconds using a sliding window approach with a shifting size of 10 milliseconds. The LLD features were extracted from these segments. For each recording, we further computed the mean of each LLD feature to capture its high-level statistical features, which were then normalized.

The cognitive status of FHS participants included assessments by NP tests. For those identified with possible cognitive impairment, NP tests were administered on average about every 1 to 2 years. When potential cognitive impairment decline was present, a clinical review was conducted by a panel with at least one neurologist and one neuropsychologist. MCI diagnosis was determined by the review panel, which required that the participant exhibit evidence of a decline in cognitive performance in 1 or more cognitive domains, have no records indicating functional decline, and do not meet the criteria for dementia [32]. The Clinical Dementia Rating scale [33] was used to quantify the severity of impairment. In all, 2 outcomes were considered in this study. The prevalent MCI cases were subjects who were diagnosed with MCI before or at the time when the voice was recorded. The incident MCI cases were all subjects who were cognitively intact at baseline but were diagnosed with MCI during the follow-up.

To compare the difference between demographics and standard NP test scores in MCI and normal control groups, Wilcoxon rank sum test was used for continuous variables [34]. The chi-square test was used to compare differences in frequencies for categorical variables [35]. Log transformations were applied for NP tests with skewed distributions to normalize them. Normalized values of NP tests and acoustic features were used in the analysis. Linear mixed-effects models were used to quantify the association between each acoustic feature and NP tests [36].

A set of acoustic composite scores was generated by regressing each NP test against the group of acoustic features that were significantly associated with each NP test. The acoustic composite score is a weighted combination of acoustic features. The weight of each acoustic feature in the composite score was derived by training a linear mixed-effects effect model. For participant i, the acoustic composite score of an NP test is defined as

where m is the number of acoustic features significantly associated with the NP test, α_j is the estimate of effect size for the acoustics feature j derived from the linear mixed-effects effect model, and V_ij is the normalized acoustics feature j for participant i. The association between normalized acoustic composite scores with corresponding NP tests was assessed by linear mixed-effects models.

The association of normalized acoustic composite scores with prevalent MCI was assessed by logistic regression models. Based on the regression coefficients, the odds ratios (ORs) and 95% CIs were estimated.

To determine the relationship between acoustic composite scores and incident MCI, participants whose age at the voice recording was <60 years (n=2718) and those with prevalent MCI (n=222) were excluded. The first observation of each participant was included in this analysis. The association between acoustic composite scores with incident MCI was quantified by Cox proportional hazards models (censored at the last date of contact or death) [37]. All models were adjusted for age, sex, and education. Bonferroni correction was used to adjust for multiple tests.

We further evaluated the added predictability of the acoustic composite score for incident MCI. Receiver operating characteristic (ROC) analysis was performed to estimate the area under the curve (AUC) using a random forest model. A baseline model was constructed using age, sex, and education as predictors. A second model was constructed using these predictors and additional acoustic composite scores that were found to be significantly related to specific NP tests. The mean AUC of 10-fold cross-validation was computed for each model for comparison. We also performed a secondary analysis by including NP tests and clinical risk factors in the prediction of incident MCI. The statistical analyses were performed using Python software (version 3.9.7; Python Software Foundation).

Relating acoustic features with NP test performance is potentially a novel way for screening at the preclinical stages of AD and other dementias. This paper clarifies the relationship between comprehensive acoustic features and NP test performance on large cohort data. Representations relative spectra–style filtered auditory spectrum (spectral), MFCC (cepstral), and magnitude of spectral features (spectral) are 3 categories of acoustic features that were significantly associated with NP test performance. Representations relative spectra–style filtered auditory spectrum is a filtered representation of an audio signal that is robust to additive and convolutional noise [39]. MFCC is a standardized technique for audio feature extraction [40]. It helps in reducing the frequency information of the input speech signal into coefficients, which represent audio based on the perception of human auditory systems. Prior studies have detected changes of these features in people with neurodegenerative processes [41-43]. The acoustic composite score generated for each NP test was a linear combination of LLD features, which are clinically easily interpretable. As stated in the results above, 4 acoustic composite scores were significantly associated with prevalent MCI, and 7 were also found to be significantly associated with incident MCI. Furthermore, the score corresponding to TrailsB test is significantly associated with both prevalent MCI and incident MCI.

Results could expand current evidence regarding the predictive ability of digital voice on MCI that are critical to monitor early cognitive decline. The added predictive ability of acoustic features was evaluated by constructing random forest models with baseline features and additional acoustic composite scores. The model with baseline features and 7 acoustic composite scores corresponding to LMi, VRi, VRd, VRr, SIM, TrailsB, and HVOT tests could achieve an AUC of 0.755 for incident MCI prediction. Monitoring acoustic features outside of the clinical settings offers a more convenient way to aid in the assessment of cognitive health than traditional methods. Increasing evidence suggests that the human voice can be a predictor of cognitive decline before a clinical diagnosis of AD is made [44]. It has been used to screen for MCI [45], dementia [46], and other neurodegenerative diseases such as Parkinson [47] and Huntington disease [48] because of its ease of administration and clinical assessment capability. Moreover, the easy acquisition of voice in daily life makes it an ideal measure for long-term monitoring of cognitive status. However, there is a lack of research about the relationship between acoustic features and NP tests that reflect multiple cognitive functions. Our study could provide some construct validity for this point. In this study, we recorded voice for NP tests that require verbal responses. Although some NP tests do not require verbal responses, these tests might tap some cognitive domains similar to those that require verbal responses. We therefore included these tests as well to capture potential application of acoustic characteristics to assess different cognitive domains. Each NP test might require multiple cognitive domains to complete, which might be shared with other NP tests with subtle differences. Given the rich information from human voice, our study suggests that acoustic features might serve as a new data modality to test this nuance.

Notably, the association between acoustic features and a standard epidemiologic NP test procedure was examined based on participants from a community-based cohort with a diverse range of ages and health conditions. The large volume of voice data provides a more robust representation of participants. Each voice recording lasts, on average, around an hour and contains a wealth of information. The longitudinal collection of data provides a great opportunity to assess the cognitive health of participants throughout the entire course of the disease and prospectively reveals a temporal relationship between acoustic features and MCI. It is worth to noting that 4 of the acoustic composite scores (acoustic_LMr, acoustic_TrailsB, acoustic_FAS, and acoustic_CNT_Animal) were significantly associated with prevalent MCI, but 7 acoustic composite scores (acoustic_LMi, acoustic_VRi, acoustic_VRd, acoustic_VRr, acoustic_SIM, acoustic_TrailsB, and acoustic_HVOT) were associated with incident MCI. It seems that the voice characteristics differentiating prevalent MCI cases from patients who are still cognitively intact are different from the voice characteristics that are predictive of future risk of cognitive impairment. Future research is needed to further investigate the potential mechanisms that underlie these features and help to account for the MCI prevalence and incidence difference. Further, this study found differences in acoustic features between TrailsA and TrailsB, which provides confirmatory evidence that acoustic features are differential for different cognitive domains. TrailsA, as a measure of simple attention compared to the more complex executive functions measured by TrailsB [49], would be expected to have different acoustic features that would be aligned with motor control and perceptual complexity [50] in the latter and not the former. These differential results suggest that acoustic features might provide a way to detect such subtle differences across cognitive domains. The patterns of acoustic features that are accurately representative of the comprehensive range of cognitive domains will be further explored in future studies.

This study also has some limitations. First, the use of NP tests to diagnose MCI may have led to some circularity and an overestimation of the diagnosis performance [32]. Second, despite that diagnoses are arrived at through a careful adjudication process, there may be some misclassification of MCI. Third, although the FHS collected the voice recordings in a well-controlled environment, there might still be some other factors affecting the quality of voice that were not taken into account. Fourth, this study did not consider linguistic features, which has been shown to be effective in predicting cognitive status. Although we recognize that the inclusion of linguistic features might further improve the prediction of incident MCI, we chose to focus on acoustic features because they are much more generalizable to a broader population, including potentially to other languages. Linguistic features are much more likely to be biased by language, culture, and education. Finally, FHS participants were mostly of European ancestry and English speakers; therefore, the applicability of our findings to populations of another race and ethnicity needs to be examined.

We examined the association of acoustic features with specific cognitive functions—prevalent and incident MCI—in a large community-based population. Overall, this study’s establishment of a relationship between MCI risk and human voice features provides foundational evidence for an alternative cognitive assessment approach that is cost-effective and easy to administer for detecting cognition-related disorders. Multiple acoustic features were significantly associated with NP test performance and MCI and could be potentially used as a digital biomarker for early cognitive impairment monitoring.