Look Both Ways: Self-supervising Driver Gaze Estimation and Road Scene Saliency

Isaac, Kasahara,; Stent, Simon; Park, Hyun Soo

doi:10.1007/978-3-031-19778-9_8

Cited by 16 publications

(4 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The first such dataset, 3DSS [38], was recorded in a video game environment. More recent ones are captured either on the road with drivers wearing head-mounted eye trackers (e.g., DR(eye)VE [14] and LBW [43]) or use pre-recorded naturalistic footage that human subjects can view in the lab (e.g., BDD-A [31] and DADA-2000 [44]). However, task-relevant information that can be effectively used for training and evaluation is usually not available or incomplete.…”

Section: Related Workmentioning

confidence: 99%

Do They Want to Cross? Understanding Pedestrian Intention for Behavior Prediction

Kotseruba

Rasouli

Tsotsos

2020

2020 IEEE Intelligent Vehicles Symposium (IV)

View full text Add to dashboard Cite

Understanding what drivers look at is important for many applications, including driver training, monitoring, and assistance, as well as self-driving. Traditionally, factors affecting human visual attention have been divided into bottomup (involuntary attraction to salient regions) and top-down (task-and context-driven). Although both play a role in drivers' gaze allocation, most of the existing modeling approaches apply techniques developed for bottom-up saliency and do not consider task and context influences explicitly. Likewise, common driving attention benchmarks lack relevant task and context annotations. Therefore, to enable analysis and modeling of these factors for drivers' gaze prediction, we propose the following: 1) address some shortcomings of the popular DR(eye)VE dataset and extend it with per-frame annotations for driving task and context; 2) benchmark a number of baseline and SOTA models for saliency and driver gaze prediction and analyze them w.r.t. the new annotations; and finally, 3) a novel model that modulates drivers' gaze prediction with explicit action and context information, and as a result significantly improves SOTA performance on DR(eye)VE overall (by 24% KLD and 89% NSS) and on a subset of action and safety-critical intersection scenarios (by 10-30% KLD). Extended annotations, code for model and evaluation will be made publicly available.

show abstract

Section: Related Workmentioning

confidence: 99%

Do They Want to Cross? Understanding Pedestrian Intention for Behavior Prediction

Kotseruba

Rasouli

Tsotsos

2020

2020 IEEE Intelligent Vehicles Symposium (IV)

View full text Add to dashboard Cite

show abstract

“…The proposed method, based on the full image (environment and face) or the full set of features (facial and Go-CaRD features; as illustrated in Figure 2 in [ 40 ]), outperformed other DL models, such as InceptionV3, ResNet50, VGG16, and VGG19. Recently, Kasahara et al [ 41 ] presented a new dataset, called “Look Both Ways”, which contains synchronized video of both driver faces and the forward road scene for gaze estimation and road scene saliency. The Look Both Ways dataset contains 123,297 synchronized driver face and stereo scene images with ground truth 3D gaze, which were collected from 6.8 h of free driving on public roads by 28 drivers.…”

Section: Driver Gaze Analysismentioning

confidence: 99%

Comprehensive Assessment of Artificial Intelligence Tools for Driver Monitoring and Analyzing Safety Critical Events in Vehicles

Yang,

Ridgeway,

Miller

et al. 2024

Sensors

View full text Add to dashboard Cite

Human factors are a primary cause of vehicle accidents. Driver monitoring systems, utilizing a range of sensors and techniques, offer an effective method to monitor and alert drivers to minimize driver error and reduce risky driving behaviors, thus helping to avoid Safety Critical Events (SCEs) and enhance overall driving safety. Artificial Intelligence (AI) tools, in particular, have been widely investigated to improve the efficiency and accuracy of driver monitoring or analysis of SCEs. To better understand the state-of-the-art practices and potential directions for AI tools in this domain, this work is an inaugural attempt to consolidate AI-related tools from academic and industry perspectives. We include an extensive review of AI models and sensors used in driver gaze analysis, driver state monitoring, and analyzing SCEs. Furthermore, researchers identified essential AI tools, both in academia and industry, utilized for camera-based driver monitoring and SCE analysis, in the market. Recommendations for future research directions are presented based on the identified tools and the discrepancies between academia and industry in previous studies. This effort provides a valuable resource for researchers and practitioners seeking a deeper understanding of leveraging AI tools to minimize driver errors, avoid SCEs, and increase driving safety.

show abstract

“…The human gaze is inherently implicit and poses a challenge for objective measurement, making gaze annotation complex in gaze data collection. Some methods capture the human gaze through intrusive devices such as eye-tracking glasses [13,19]. However, the eye-tracking glasses have a notable impact on the quality of the captured facial images.…”

Section: Gaze Data Collectionmentioning

confidence: 99%

“…In-vehicle gaze dataset usually defines different region such as windshield and left/right mirror in vehicles and perform gaze zone classification [12,14,16,18,23,30,34]. Kasahara et al [19] collects an in-vehicle gaze dataset but subjects are required to wear eye-tracking glasses, which means the dataset is not applicable in the real world. Our gaze collection system does not require dedicated devices and produces natural face images.…”

Section: Datasetsmentioning

confidence: 99%

Comprehensive Evaluation Method of Arrangement, Scheduling and Route of Multi-vehicle

Xing

Liu

Jin

et al. 2022

2022 34th Chinese Control and Decision Conference (CCDC)

View full text Add to dashboard Cite

Driver's eye gaze holds a wealth of cognitive and intentional cues crucial for intelligent vehicles. Despite its significance, research on in-vehicle gaze estimation remains limited due to the scarcity of comprehensive and wellannotated datasets in real driving scenarios. In this paper, we present three novel elements to advance in-vehicle gaze research. Firstly, we introduce IVGaze, a pioneering dataset capturing in-vehicle gaze, collected from 125 subjects and covering a large range of gaze and head poses within vehicles. In this dataset, we propose a new visionbased solution for in-vehicle gaze collection, introducing a refined gaze target calibration method to tackle annotation challenges. Second, our research focuses on in-vehicle gaze estimation leveraging the IVGaze. In-vehicle face images often suffer from low resolution, prompting our introduction of a gaze pyramid transformer that leverages transformer-based multilevel features integration. Expanding upon this, we introduce the dual-stream gaze pyramid transformer (GazeDPTR). Employing perspective transformation, we rotate virtual cameras to normalize images, utilizing camera pose to merge normalized and original images for accurate gaze estimation. GazeDPTR shows stateof-the-art performance on the IVGaze dataset. Thirdly, we explore a novel strategy for gaze zone classification by extending the GazeDPTR. A foundational tri-plane and project gaze onto these planes are newly defined. Leveraging both positional features from the projection points and visual attributes from images, we achieve superior performance compared to relying solely on visual features, substantiating the advantage of gaze estimation. Our project is available at https://yihua.zone/work/ivgaze.1 Our work focuses on gaze direction estimation. We abbreviate gaze direction as gaze in the rest.

show abstract

Look Both Ways: Self-supervising Driver Gaze Estimation and Road Scene Saliency

Cited by 16 publications

References 45 publications

Do They Want to Cross? Understanding Pedestrian Intention for Behavior Prediction

Do They Want to Cross? Understanding Pedestrian Intention for Behavior Prediction

Comprehensive Assessment of Artificial Intelligence Tools for Driver Monitoring and Analyzing Safety Critical Events in Vehicles

Comprehensive Evaluation Method of Arrangement, Scheduling and Route of Multi-vehicle

Contact Info

Product

Resources

About