X-AWARE: ConteXt-AWARE Human-Environment Attention Fusion for Driver Gaze Prediction in the Wild

Abstract

Reliable systems for automatic estimation of the driver's gaze are crucial for reducing the number of traffic fatalities and for many emerging research areas aimed at developing intelligent vehiclepassenger systems. Gaze estimation is a challenging task, especially in environments with varying illumination and reflection properties. Furthermore, there is wide diversity with respect to the appearance of drivers' faces, both in terms of occlusions (e. g., vision aids) and cultural/ethnic backgrounds. For this reason, analysing the face along with contextual information ś for example, the vehicle cabin environment ś adds another, less subjective signal towards the design of robust systems for passenger gaze estimation. In this paper, we present an integrated approach to jointly model different features for this task. In particular, to improve the fusion of the visually captured environment with the driver's face, we have developed a contextual attention mechanism, X-AWARE , attached directly to the output convolutional layers of InceptionResNetV2 networks. In order to showcase the effectiveness of our approach, we use the Driver Gaze in the Wild dataset, recently released as part of the Eighth Emotion Recognition in the Wild Challenge (EmotiW) challenge. Our best model outperforms the baseline by an absolute of 15.03 % in accuracy on the validation set, and improves the previously best reported result by an absolute of 8.72 % on the test set. CCS CONCEPTS• Computing methodologies → Computer vision; Scene understanding; • Human-centered computing → Human computer interaction (HCI).

XAIR: A Framework of Explainable AI in Augmented Reality

“…The integration of sensors enables AR systems to understand users' current states [99,194,204] and their environment [139,153] to provide a variety of intelligent functionalities. For example, AR could infer user intent [14] and provide contextual recommendations for daily activities (e.g., recipe recommendations when a user opens the fridge during lunch) [15,118,122].…”

“…User State. The sensors that could be integrated within future HMDs would empower an AR system to have a rich, instant understanding of user's state, such as activities (IMU [86,219], camera [80,128,194,201], microphone [103,218,229,230]), cognitive load (eye tracking [71,104,238], EEG [20,224]), attention (eye tracking [56,99,204,231], IMU [123], EEG [213]), emotion (facial tracking [233,236], EEG [202,216]) and potential intent (the fusion of multiple sensors and low-level intelligence [14,111,211]). Depending on a user's state, the design of explanations could be different.…”

E2DR: A Deep Learning Ensemble-Based Driver Distraction Detection with Recommendations Model

“…A gaze estimation model called X-Aware is introduced in [7] to analyze the driver's face along with contextual information. The model visually improves the fusion of the captured environment of the driver's face, where the contextual attention mechanism is directly attached to the output of convolutional layers of the InceptionResNetV2 networks.…”

Attention for Vision-Based Assistive and Automated Driving: A Review of Algorithms and Datasets

“…Deep learning models have the advantage of combining feature extraction and classification steps. Instead of the explicit processing pipeline described above, a single convolutional neural network (CNN) pre-trained on the image classification task can be used to classify cropped frames from driver-facing videos [80]- [83]. These CNN-based models reach high accuracy and can discriminate adjacent areas better than previous methods that relied on hand-crafted features.…”