This is an open-access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on http://www.jmir.org/, as well as this copyright and license information must be included.
Driving a car is a complex instrumental activity of daily living and driving performance is very sensitive to cognitive impairment. The assessment of driving-relevant cognition in older drivers is challenging and requires reliable and valid tests with good sensitivity and specificity to predict safe driving. Driving simulators can be used to test fitness to drive. Several studies have found strong correlation between driving simulator performance and on-the-road driving. However, access to driving simulators is restricted to specialists and simulators are too expensive, large, and complex to allow easy access to older drivers or physicians advising them. An easily accessible, Web-based, cognitive screening test could offer a solution to this problem. The World Wide Web allows easy dissemination of the test software and implementation of the scoring algorithm on a central server, allowing generation of a dynamically growing database with normative values and ensures that all users have access to the same up-to-date normative values.
In this pilot study, we present the novel Web-based Bern Cognitive Screening Test (wBCST) and investigate whether it can predict poor simulated driving performance in healthy and cognitive-impaired participants.
The wBCST performance and simulated driving performance have been analyzed in 26 healthy younger and 44 healthy older participants as well as in 10 older participants with cognitive impairment. Correlations between the two tests were calculated. Also, simulated driving performance was used to group the participants into good performers (n=70) and poor performers (n=10). A receiver-operating characteristic analysis was calculated to determine sensitivity and specificity of the wBCST in predicting simulated driving performance.
The mean wBCST score of the participants with poor simulated driving performance was reduced by 52%, compared to participants with good simulated driving performance (
When selecting a 75% test score as the cutoff, the novel test has 83% sensitivity, 70% specificity, and 81% efficiency, which are good values for a screening test. Overall, in this pilot study, the novel Web-based computer test appears to be a promising tool for supporting clinicians in fitness-to-drive assessments of older drivers. The Web-based distribution and scoring on a central computer will facilitate further evaluation of the novel test setup. We expect that in the near future, Web-based computer tests will become a valid and reliable tool for clinicians, for example, when assessing fitness to drive in older drivers.
Driving a car is a very challenging instrumental activity of daily living that requires the integration of high-level cognition, vision, and motor function [
Since driving is a complex activity, driving performance is very sensitive to cognitive impairment [
On-the-road testing (ORT) has been suggested as being a reasonable proxy measure for naturalistic driving in older adults with a range of cognitive impairments [
That is why more recently, driving simulators (DS) have been recommended as a proxy measure for naturalistic driving and they have been introduced to assess fitness to drive of older drivers with and without cognitive impairment [
In a recent study, Rockwood et al [
In this pilot study, we investigate whether or not the novel wBCST correlates with DS performance and whether it is able to differentiate between participants with poor and good simulated driving performance. To have a broad and diverse test population, we recruited younger and older healthy participants as well as older participants with cognitive impairment for this study. Hence, this paper first describes the novel wBCST and the DS used, followed by a correlation analysis and a receiver operating characteristics (ROC) [
Thirty healthy younger adults (age 22-40 years), 60 healthy older adults (age >50 years), and 15 older (age >50 years) participants with cognitive impairment (Montreal Cognitive Assessment Score [MoCA] <26) [
A literature review of the most important driving-relevant cognitive functions and how they are affected by cognitive impairment was conducted [
Subtest 1 measures selective attention with a visual scene consisting of a simplified street in the center of the screen (
Older participant during the wBCST evaluation. A 24-inch monitor (a) is used to present the test material and participants interact with the system via a steering wheel (b) and foot pedal (c). Each subtest uses similar graphical objects as shown in the screenshots (d-h).
A commercially available high-fidelity fixed-frame driving simulator (F12PI-3/A88, Foerst GmbH) with a custom-built virtual driving circuit was used to measure DS performance. A virtual scene was projected by three projectors (Ultra-Short focus LCD projectors, Sanyo) with 1024x786 pixel resolution onto three projection screens (1.8 x 1.4 m) that were installed in front of the driver. DS components utilized by the participants were steering wheel, brake and gas pedals, rear and side mirrors, and turn signals (
Participants performed a familiarization run (three minutes) to get used to the handling of the simulator and a test drive (six minutes) during which data were recorded. Once the end of the track was reached, the car stopped automatically. If a participant felt uncomfortable, the DS was stopped. Primary outcome measure of the driving simulator was the number of errors,
High-fidelity fixed-frame driving simulator with younger test subject. The steering wheel, cockpit, and parts of two projection screens with the virtual driving screen are shown.
The results of the wBCST were ranked and scored. Thus, the score,
Pearson product-moment correlations, step-wise regressions, and associated tests of significance were calculated across
In a second step, the number of errors
The data of 80 participants were included in the data analysis. Ten were assigned, based on the number of errors in the DS using mean value + 1 SD cutoff, to the poor DS performance group (
User statistics of all participants and the two subgroups.
|
All Participants, |
Participants with good DSa performance, |
Participants with poor DS performance, |
Significance of group difference |
Young (<40 years), n | 26 | 26 | 0 |
|
Older (>50 years), without cognitive impairment (MoCAb≥26), n | 44 | 38 | 6 |
|
Older (>50 years), with cognitive impairment (MoCA<26), n | 10 | 6 | 4 |
|
MoCA, score (SD) | 28.2 (2.9) | 28.5 (2.8) | 26.2 (3.1) |
|
TMTc A, seconds (SD) | 28.0 (14.4) | 26.9 (14.2) | 35.7 (14.1) |
|
TMT B, seconds (SD) [ |
71.6 (52.5) | 68.8 (53.2) | 91.2 (45.7) |
|
CDT, score (SD) | 6.2 (1.9) | 6.5 (1.6) | 4.3 (3.1) |
|
Timed-up-and-go-test, seconds (SD) | 6.9 (3.2) | 6.8 (2.9) | 7.5 (4.7) |
|
aDS: driving simulator
bMoCA: Montreal Cognitive Assessment
cTMT: trail making test (A and B)
dCDT: clock drawing test
The ranked normalized wBCST performances of the two groups are represented in
The ROC curve for using
Correlation analysis of the wBCST and the DS. The table shows the Pearson product-moment correlation coefficient
Receiver operating characteristic (ROC) curve analysis: possible cutoff values and corresponding sensitivity, specificity, and efficiency.
Cutoff | Sensitivity | Specificity | Efficiency |
0.88 | 0.94 | 0.30 | 0.86 |
0.84 | 0.91 | 0.30 | 0.84 |
0.83 | 0.89 | 0.30 | 0.81 |
0.81 | 0.86 | 0.30 | 0.79 |
0.77 | 0.84 | 0.50 | 0.80 |
0.75 | 0.83 | 0.70 | 0.81 |
0.73 | 0.80 | 0.70 | 0.79 |
0.68 | 0.77 | 0.70 | 0.76 |
0.66 | 0.74 | 0.70 | 0.74 |
0.65 | 0.71 | 0.70 | 0.71 |
0.64 | 0.69 | 0.70 | 0.69 |
0.63 | 0.69 | 0.90 | 0.71 |
0.61 | 0.66 | 0.90 | 0.69 |
0.57 | 0.63 | 0.90 | 0.66 |
0.56 | 0.60 | 0.90 | 0.64 |
0.53 | 0.57 | 0.90 | 0.61 |
0.52 | 0.54 | 0.90 | 0.59 |
0.51 | 0.53 | 0.90 | 0.58 |
Web-based Bern Cognitive Screening Test (wBCST) performance of the group with good simulated driving performance (n=70) and group with poor simulator performance (n=10). All values are normalized and ranked. The score S_wBCST is the mean value of the subscores S (1…5). Subtest 3 is represented with two entries, S_3p for the peripheral recognition task and S_3c for the central steering task. Bars indicate the standard error.
Receiver operating characteristics (ROC) curve for using S_wBCST to predict SD driving performance, respectively to differentiate between subjects with good and poor DS performance. The thin gray line is the empirical curve, the solid black line is the smoothed (Gaussian-based) curve, and the dotted diagonal line indicates no discrimination (50% chance).
When using the criteria proposed by Cohen [
The group differences in the wBCST test and its subscores are all significant, which is a prerequisite for using the wBCST to predict DS performance. The AUC of the ROC curve is slightly larger than 0.80, which is generally considered a good test [
In the present study, the overall test result,
Overall, the novel wBCST was very well accepted by the test population and there were no drop-outs in the wBCST. This is in contrast to the DS with 19 drop-outs (19.1%) due to simulator sickness (18 drop-outs) and technical problems (1 drop-out). Since it is Web-based, the distribution of the test software to different test-sites will be fairly easy and with the central scoring algorithm, data consistency among the different centers can be ensured, which will facilitate validation studies in larger populations.
The online instructions of the test procedure take about 5 minutes. We did not observe difficulties of the test persons to understand the task, except two cognitively impaired test subjects wanted to go through the instructions twice. We concluded that the instructions are clear, but participants should have the option to repeat the instructions.
The steering wheel and the foot pedal are fairly cheap accessories, but to further improve accessibility of the novel test it would be beneficial if the test could also be used with keyboard and mouse. We observed that steering wheel and foot pedal seem to increase the face-validity of the tests, but we would expect that the measured cognitive functions should be independent from the input modality. This could be investigated in a future study.
This pilot study has some limitations that need to be mentioned. One is the large number of drop-outs in the DS that might create a selection bias, since it cannot be excluded that people with poor driving performance might be more prone to simulator sickness. Although the current literature on simulator sickness suggests other contributing factors (eg, age, gender), this cannot be excluded [
In this pilot study, the novel wBCST looks like a promising test to support clinicians in fitness-to-drive assessments of older drivers. The Web-based distribution and the scoring on a central computer will facilitate further evaluation of the novel test setup. In its current form, the program requires local installation on a client computer in the physician’s office. This is currently not considered a disadvantage, but, of course, one could transfer the test program to run within a Web browser, which would not require local installation. The hardware requirements of the wBCST are very minor and include an office-type personal computer with Windows 7 operating system and a simple steering wheel (eg, Driving Force GT, Logitec Inc). Overall, when considering disadvantages of DS (costs, simulator sickness, space requirements), the authors believe that in many clinical environments the wBCST is better suited to support physicians in fitness-to-drive assessments than a DS. That is why we expect that in the near future, Web-based computer tests will become a valuable and usable tool for fitness-to-drive assessment in older drivers.
Screencast with animated images from the experimental setup (.wmv File, 44 MB).
Screencast with animated images from the experimental setup (.mp4 File, 92 MB).
area under the curve
clock drawing test
driving simulator
Montreal Cognitive Assessment
on-the-road testing
receiver operating characteristic curve
trail making tests A/B
Web-based Bern Cognitive Screening Test
This work has been supported in part by the Swiss foundation for traffic safety (Fonds für Verkehrssicherheit). The authors would like to thank the members of the expert group, namely A Boss-Skupnjak and Prof Dr HP Lindenmann for their valuable input to the data analysis.
None declared.