Distracted and Drowsy Driving Modeling Using Deep Physiological Representations and Multitask Learning

Abstract

In this paper, we investigated various physiological indicators on their ability to identify distracted and drowsy driving. In particular, four physiological signals are being tested: blood volume pulse (BVP), respiration, skin conductance and skin temperature. Data were collected from 45 participants, under a simulated driving scenario, through different times of the day and during their engagement on a variety of physical and cognitive distractors. We explore several statistical features extracted from those signals and their efficiency to discriminate between the presence or not of each of the two conditions. To that end, we evaluate three traditional classifiers (Random Forests, KNN and SVM), which have been extensively applied by the related literature and we compare their performance against a deep CNN-LSTM network that learns spatio-temporal physiological representations. In addition, we explore the potential of learning multiple conditions in parallel using a single machine learning model, and we discuss how such a problem could be formulated and what are the benefits and disadvantages of the different approaches. Overall, our findings indicate that information related to the BVP data, especially features that describe patterns with respect to the inter-beat-intervals (IBI), are highly associates with both targeted conditions. In addition, features related to the respiratory behavior of the driver can be indicative of drowsiness, while being less associated with distractions. Moreover, spatio-temporal deep methods seem to have a clear advantage against traditional classifiers on detecting both driver conditions. Our experiments show, that even though learning both conditions jointly can not compete directly to individual, task-specific CNN-LSTM models, deep multitask learning approaches have a great potential towards that end as they offer the second best performance on both tasks against all other evaluated alternatives in terms of sensitivity, specificity and the area under the receiver operating characteristic curve (AUC).

Extended-Range Prediction Model Using NSGA-III Optimized RNN-GRU-LSTM for Driver Stress and Drowsiness

“…Nature of dataset: The existing works [ 14 , 15 , 16 , 17 , 18 ] utilized data from a simulated environment, in which the data may not reflect the real-world environment. Our work considered a real-world dataset, which verifies the validity of the prediction model in real-world deployment.…”

“…Dataset: 11–45 participants contributed the datasets in existing works [ 14 , 15 , 16 , 17 , 18 ]. Although the dataset in our work included 108 participants, all of them are small-scale datasets.…”

“…Features: The feature extraction approaches can be categorized into image-based [ 14 , 15 , 18 ] and biometric-signal-based approaches [ 16 , 17 ] (and ours). Statistics or signal processing techniques were involved to compute the features in [ 14 , 17 , 18 ] (and ours), whereas deep learning was adopted in [ 15 , 16 ]. One study reported on the issue of data quality for images and EEG [ 51 ].…”

“…Methodology: Two types of architectures—namely, single core (one core algorithm) [ 14 , 15 , 16 , 17 , 18 ] and hybrid (multiple core algorithms) (our proposal that links the RNN, GRU, and LSTM algorithms) approaches—were adopted. As demonstrated in Section 3.1 , Section 3.2 and Section 3.3 , the hybrid approach is superior for enhancing the performance as it benefits from the advantages of multiple algorithms.…”

“…An accuracy of 75% was achieved for a prediction of 3–5 s in advance. Furthermore, CNN-LSTM was adopted in [ 16 ], with multiple inputs using the blood volume pulse, skin temperature, skin conductance, and respiration of the drivers. The results of a 8 s in-advance prediction showed an average recall, specificity, and sensitivity of 82%, 71%, and 93%, respectively.…”