Driver Emotion and Fatigue State Detection Based on Time Series Fusion

Abstract

Studies have shown that driver fatigue or unpleasant emotions significantly increase driving risks. Detecting driver emotions and fatigue states and providing timely warnings can effectively minimize the incidence of traffic accidents. However, existing models rarely combine driver emotion and fatigue detection, and there is space to improve the accuracy of recognition. In this paper, we propose a non-invasive and efficient detection method for driver fatigue and emotional state, which is the first time to combine them in the detection of driver state. Firstly, the captured video image sequences are preprocessed, and Dlib (image open source processing library) is used to locate face regions and mark key points; secondly, facial features are extracted, and fatigue indicators, such as driver eye closure time (PERCLOS) and yawn frequency are calculated using the dual-threshold method and fused by mathematical methods; thirdly, an improved lightweight RM-Xception convolutional neural network is introduced to identify the driver’s emotional state; finally, the two indicators are fused based on time series to obtain a comprehensive score for evaluating the driver’s state. The results show that the fatigue detection algorithm proposed in this paper has high accuracy, and the accuracy of the emotion recognition network reaches an accuracy rate of 73.32% on the Fer2013 dataset. The composite score calculated based on time series fusion can comprehensively and accurately reflect the driver state in different environments and make a contribution to future research in the field of assisted safe driving.

Scaling effectivity in manifold methodologies to detect driver’s fatigueness and drowsiness state

“…Arefnezhad et al [33] have presented a study of risk of fatigueness during automated driving and used difference of PERCLOS for analyzing the proportion of eye closure in order to determine driver's fatigueness. Adoption of eye closure is also reported in work of Dzuida et al [34] and Shang et al [35] while similar trend of work is also reported by Chen et al [36] where facial detection is initially carried out using Adaboost while tracing of facial movement is done by Kalman filter. Further facial landmarks were detected using regression tree of cascaded form followed by using backpropagation neural network for training.…”

How to Prevent Drivers before Their Sleepiness Using Deep Learning-Based Approach

“…The driver does not like to use invasive methods. [27] 0.981 Video 9 Long et al [28] 0.9 Video 2 Arunasalam et al [31] N/A Video+ECG 3 Xiao et al [47] 0.991 Video 2 Shang et al [48] 0.733 Video 4…”