Simulator Pre-Screening of Underprepared Drivers Prior to Licensing On-Road Examination: Clustering of Virtual Driving Test Time Series Data

Abstract

Background A large Midwestern state commissioned a virtual driving test (VDT) to assess driving skills preparedness before the on-road examination (ORE). Since July 2017, a pilot deployment of the VDT in state licensing centers (VDT pilot) has collected both VDT and ORE data from new license applicants with the aim of creating a scoring algorithm that could predict those who were underprepared. Objective Leveraging data collected from the VDT pilot, this study aimed to develop and conduct an initial evaluation of a novel machine learning (ML)–based classifier using limited domain knowledge and minimal feature engineering to reliably predict applicant pass/fail on the ORE. Such methods, if proven useful, could be applicable to the classification of other time series data collected within medical and other settings. Methods We analyzed an initial dataset that comprised 4308 drivers who completed both the VDT and the ORE, in which 1096 (25.4%) drivers went on to fail the ORE. We studied 2 different approaches to constructing feature sets to use as input to ML algorithms: the standard method of reducing the time series data to a set of manually defined variables that summarize driving behavior and a novel approach using time series clustering. We then fed these representations into different ML algorithms to compare their ability to predict a driver’s ORE outcome (pass/fail). Results The new method using time series clustering performed similarly compared with the standard method in terms of overall accuracy for predicting pass or fail outcome (76.1% vs 76.2%) and area under the curve (0.656 vs 0.682). However, the time series clustering slightly outperformed the standard method in differentially predicting failure on the ORE. The novel clustering method yielded a risk ratio for failure of 3.07 (95% CI 2.75-3.43), whereas the standard variables method yielded a risk ratio for failure of 2.68 (95% CI 2.41-2.99). In addition, the time series clustering method with logistic regression produced the lowest ratio of false alarms (those who were predicted to fail but went on to pass the ORE; 27.2%). Conclusions Our results provide initial evidence that the clustering method is useful for feature construction in classification tasks involving time series data when resources are limited to create multiple, domain-relevant variables.

Examining virtual driving test performance and its relationship to individuals with HIV-associated neurocognitive disorders

“…These channels were ultimately converted into unique variables (e.g., min, max, mean, median, standard deviation of each channel) and further organized into global (across the entire VDT route) and zone-specific variables (within 22 unique segments of the driving route). In total, 2,921 time-series-derived variables were computed for each study participant’s VDT session (previously described in Grethlein et al, 2020 ). In addition to the variables described above, 8 higher-order errors (e.g., counts of collisions, red light errors, stop sign errors, navigational errors, etc.)…”

“…In a novel application, this research program aims to utilize a well-established virtual driving test (VDT) platform ( Walshe et al, 2020 ) that has been validated to predict on-road performance ( Winston et al, 2019 ; Grethlein et al, 2020 ) as a probe to screen for neurocognitive impairment among PWH. The currently used HAND screening tools (e.g., IHDS, MoCA, MMSE, SSQ, CAT-Rapid) may have minimal direct costs associated with them (e.g., cost to purchase license for use, cost to purchase equipment, if any); however, there is an indirect cost of administering these tests in routine clinical practice (e.g., trained staff labor).…”