A Driving Performance Forecasting System Based on Brain Dynamic State Analysis Using 4-D Convolutional Neural Networks

Abstract

Vehicle accidents are the primary cause of fatalities worldwide. Most often, experiencing fatigue on the road leads to operator errors and behavioral lapses. Thus, there is a need to predict the cognitive state of drivers, particularly their fatigue level. Electroencephalography (EEG) has been demonstrated to be effective for monitoring changes in the human brain state and behavior. Thirty-seven subjects participated in this driving experiment and performed a perform lane-keeping task in a visual-reality environment. Three domains, namely, frequency, temporal and 2D spatial information, of EEG channel location were comprehensively considered. A 4D convolutional neural network (4D CNN) algorithm was then proposed to associate all information from the EEG signals and the changes in the human state and behavioral performance. 4D CNN achieves superior forecasting performance over 2D CNN, 3D CNN and shallow networks. The results showed a 3.82% improvement in the RMSE, a 3.45% improvement in the error rate, and a 11.98% improvement in the correlation coefficient with 4D CNN compared with 3D CNN. The 4D CNN algorithm extracts the significant theta and alpha activations in the frontal and posterior cingulate cortices under distinct fatigue levels. This work contributes to enhancing our understanding of deep learning methods in the analysis of EEG signals. We even envision that deep learning might serve as a bridge between translation neuroscience and further real-world applications.

Extended-Range Prediction Model Using NSGA-III Optimized RNN-GRU-LSTM for Driver Stress and Drowsiness

“…Nature of dataset: The existing works [ 14 , 15 , 16 , 17 , 18 ] utilized data from a simulated environment, in which the data may not reflect the real-world environment. Our work considered a real-world dataset, which verifies the validity of the prediction model in real-world deployment.…”

“…Dataset: 11–45 participants contributed the datasets in existing works [ 14 , 15 , 16 , 17 , 18 ]. Although the dataset in our work included 108 participants, all of them are small-scale datasets.…”

“…Features: The feature extraction approaches can be categorized into image-based [ 14 , 15 , 18 ] and biometric-signal-based approaches [ 16 , 17 ] (and ours). Statistics or signal processing techniques were involved to compute the features in [ 14 , 17 , 18 ] (and ours), whereas deep learning was adopted in [ 15 , 16 ]. One study reported on the issue of data quality for images and EEG [ 51 ].…”

“…Methodology: Two types of architectures—namely, single core (one core algorithm) [ 14 , 15 , 16 , 17 , 18 ] and hybrid (multiple core algorithms) (our proposal that links the RNN, GRU, and LSTM algorithms) approaches—were adopted. As demonstrated in Section 3.1 , Section 3.2 and Section 3.3 , the hybrid approach is superior for enhancing the performance as it benefits from the advantages of multiple algorithms.…”

“…The results of a 8 s in-advance prediction showed an average recall, specificity, and sensitivity of 82%, 71%, and 93%, respectively. Lin et al [ 17 ] presented a 4-D CNN algorithm for a 6 s in-advance prediction. The 2-D spatial information, temporal information, and frequency of the electroencephalogram (EEG) signal were extracted.…”