HammerDrive: A Task-Aware Driving Visual Attention Model

Abstract

We introduce HammerDrive, a novel architecture for task-aware visual attention prediction in driving. The proposed architecture is learnable from data and can reliably infer the current focus of attention of the driver in real-time, while only requiring limited and easy-to-access telemetry data from the vehicle. We build the proposed architecture on two core concepts: 1) driving can be modeled as a collection of sub-tasks (maneuvers), and 2) each sub-task affects the way a driver allocates visual attention resources, i.e., their eye gaze fixation. HammerDrive comprises two networks: a hierarchical monitoring network of forward-inverse model pairs for sub-task recognition and an ensemble network of task-dependent convolutional neural network modules for visual attention modeling. We assess the ability of HammerDrive to infer driver visual attention on data we collected from 20 experienced drivers in a virtual reality-based driving simulator experiment. We evaluate the accuracy of our monitoring network for sub-task recognition and show that it is an effective and light-weight network for reliable real-time tracking of driving maneuvers with above 90% accuracy. Our results show that HammerDrive outperforms a comparable stateof-the-art deep learning model for visual attention prediction on numerous metrics with ∼13% improvement for both Kullback-Leibler divergence and similarity, and demonstrate that taskawareness is beneficial for driver visual attention prediction.

Attention for Vision-Based Assistive and Automated Driving: A Review of Algorithms and Datasets

“…For example, [188], [189] use vanishing point because drivers often focus on it to get an optimal view of the road ahead [25], [193]. Other options include drivers' actions [190]- [192], current driving task [194], [195], previous fixation locations [190]- [192], and vehicle telemetry [190]- [192], [196]. Other common features include semantic segmentation maps [197]- [199] and detected objects [196], [200]- [202].…”

“…For instance, optical flow is useful for identifying the direction and magnitude of motion in the scene [198], [203]- [205]. More recently, following successful applications in video action recognition problems [206], 3D convolutional networks have become a popular choice for encoding spatiotemporal data [194], [199], [202], [207]. Some approaches use recurrent networks, combined with individually encoded frames [208], [209] or with a set of frames processed via 3D convolutional layers [197].…”

“…Two recent models, HammerDrive [194] and MEDIRL [195], demonstrate that explicitly modeling the underlying driving task is beneficial for gaze prediction performance. In HammerDrive, a separate module recognizes maneuvers (lane change and lane-keeping) from the vehicle telemetry.…”

“…3) Better Coordination Between Research Areas: Research areas covered in this survey are complementary and can benefit from coordinating their efforts. For example, taking into account driver's actions helps better predict attention [194], [195] and detect inattention [109], [258]. Likewise, gaze and appearance features are useful for detecting both distraction and drowsiness (Section IV) but relatively few works investigate these problems together [259], [260].…”

CoCAtt: A Cognitive-Conditioned Driver Attention Dataset

“…They found that fixations are geared towards obtaining the most valuable task-specific information and thus the subject may not necessarily fixate on the most salient regions in a scene [10,11]. The effect of task on visual attention has given rise to task-driven attention models, which have shown to be effective for visual attention prediction [2,8]. For example, Gao et al present the new problem of Object Importance Estimation, and propose a framework composed of a visual model and a goal model to directly incorporate the effect of driving goal into the task at hand.…”