Recognizing Frustration of Drivers From Face Video Recordings and Brain Activation Measurements With Functional Near-Infrared Spectroscopy

Abstract

Experiencing frustration while driving can harm cognitive processing, result in aggressive behavior and hence negatively influence driving performance and traffic safety. Being able to automatically detect frustration would allow adaptive driver assistance and automation systems to adequately react to a driver’s frustration and mitigate potential negative consequences. To identify reliable and valid indicators of driver’s frustration, we conducted two driving simulator experiments. In the first experiment, we aimed to reveal facial expressions that indicate frustration in continuous video recordings of the driver’s face taken while driving highly realistic simulator scenarios in which frustrated or non-frustrated emotional states were experienced. An automated analysis of facial expressions combined with multivariate logistic regression classification revealed that frustrated time intervals can be discriminated from non-frustrated ones with accuracy of 62.0% (mean over 30 participants). A further analysis of the facial expressions revealed that frustrated drivers tend to activate muscles in the mouth region (chin raiser, lip pucker, lip pressor). In the second experiment, we measured cortical activation with almost whole-head functional near-infrared spectroscopy (fNIRS) while participants experienced frustrating and non-frustrating driving simulator scenarios. Multivariate logistic regression applied to the fNIRS measurements allowed us to discriminate between frustrated and non-frustrated driving intervals with higher accuracy of 78.1% (mean over 12 participants). Frustrated driving intervals were indicated by increased activation in the inferior frontal, putative premotor and occipito-temporal cortices. Our results show that facial and cortical markers of frustration can be informative for time resolved driver state identification in complex realistic driving situations. The markers derived here can potentially be used as an input for future adaptive driver assistance and automation systems that detect driver frustration and adaptively react to mitigate it.

Estimating Cognitive Workload in an Interactive Virtual Reality Environment Using EEG

“…In contrast, implicit or passive BCI control, where the user's cognitive or affective state is passively monitored and used to affect some auxiliary aspect of the interaction (Zander and Kothe, 2011; Brouwer et al, 2015; Unni et al, 2017; Horvat et al, 2018; Ihme et al, 2018) may be better-suited for practical integration into VR systems. Such passive feedback can be designed to be less sensitive to BCI decoding errors, with the potential of being less noticeable and distracting to the user compared to decoding errors in direct BCI control of the environment.…”

Shedding light on the prefrontal correlates of mental workload in simulated driving: a functional near-infrared spectroscopy study

“…fNIRS however does not suffer from the same restrictions as fMRI and EEG regarding movement or environmental interferences and thus as well has gained more and more popularity in driving research. In the last decade amongst others, the neuronal correlates of specific driving maneuvers [24][25][26][27][28] , drowsiness and fatigue [29][30][31][32][33][34][35][36][37][38] habituation 39 and frustration during driving 40 have been examined with fNIRS. Further, several fNIRS studies have examined the neural correlates of mental workload during different driving operations.…”

Towards affect‐aware vehicles for increasing safety and comfort: recognising driver emotions from audio recordings in a realistic driving study

“…[29]). Thus, researchers have developed methods to extract indicators for emotions from video recordings of the face [30–35]. With the aforementioned introduction of voice assistants into vehicles, speech recording and analysis of prosodic features becomes more interesting for driver state assessment, because even people alone in the vehicles may use their voice more often.…”