DRER: Deep Learning–Based Driver’s Real Emotion Recognizer

Abstract

In intelligent vehicles, it is essential to monitor the driver’s condition; however, recognizing the driver’s emotional state is one of the most challenging and important tasks. Most previous studies focused on facial expression recognition to monitor the driver’s emotional state. However, while driving, many factors are preventing the drivers from revealing the emotions on their faces. To address this problem, we propose a deep learning-based driver’s real emotion recognizer (DRER), which is a deep learning-based algorithm to recognize the drivers’ real emotions that cannot be completely identified based on their facial expressions. The proposed algorithm comprises of two models: (i) facial expression recognition model, which refers to the state-of-the-art convolutional neural network structure; and (ii) sensor fusion emotion recognition model, which fuses the recognized state of facial expressions with electrodermal activity, a bio-physiological signal representing electrical characteristics of the skin, in recognizing even the driver’s real emotional state. Hence, we categorized the driver’s emotion and conducted human-in-the-loop experiments to acquire the data. Experimental results show that the proposed fusing approach achieves 114% increase in accuracy compared to using only the facial expressions and 146% increase in accuracy compare to using only the electrodermal activity. In conclusion, our proposed method achieves 86.8% recognition accuracy in recognizing the driver’s induced emotion while driving situation.

Causal affect prediction model using a facial image sequence

“…Although these methods perform well with large-scale data in the wild, they encounter limitations when used in real time. Except in cases such as posterior analysis with recorded video, most real-world applications require future decisions only with past data in real time, such as facial expression recognition for interview assistant systems [21] or driver monitoring systems [19].…”

“…CAPNet consists of a modular architecture divided into a feature extractor and a causality extractor, which allows one to learn causality well from past facial images. The feature extractor is based on the facial expression recognition (FER) model proposed by Oh et al [19], which was pretrained using the AffectNet dataset [17]. We fine-tuned this FER model with a pair of single images and corresponding labels of the Aff-Wild2 dataset [10] and used the CNN architecture of the fine-tuned FER (FER-Tuned) model as the feature extractor of CAPNet.…”

“…• We fine-tune the pretrained FER model proposed by Oh et al [19] and propose the performance of the FER-Tuned model using a current facial image as a high baseline for human affective behavior recognition.…”

Skin potential response for stress recognition in simulated urban driving

“…Driver's expressions and eye movements are recorded by near-infrared (NIR) camera sensors, and then aggressive driving behaviour is classified by a CNN. The method proposed in [21] combines both physiological (electrodermal activity) and behavioural (facial) measurements, and fuses together different data types in order to build a Sensor Fusion Emotion Recognition (SFER) system, improving the classification performance.…”

A New Event-Triggered Type-3 Fuzzy Control System for Multi-Agent Systems: Optimal Economic Efficient Approach for Actuator Activating

“…In the second class, the neuro-fuzzy controllers and machine learning approaches have been developed [13][14][15][16]. For Example, the improvement of backstepping control method by the use of FLSs is investigated in [17], and the time convergence is analyzed.…”