Automatic Visual Attention Detection for Mobile Eye Tracking Using Pre-Trained Computer Vision Models and Human Gaze

Abstract

Processing visual stimuli in a scene is essential for the human brain to make situation-aware decisions. These stimuli, which are prevalent subjects of diagnostic eye tracking studies, are commonly encoded as rectangular areas of interest (AOIs) per frame. Because it is a tedious manual annotation task, the automatic detection and annotation of visual attention to AOIs can accelerate and objectify eye tracking research, in particular for mobile eye tracking with egocentric video feeds. In this work, we implement two methods to automatically detect visual attention to AOIs using pre-trained deep learning models for image classification and object detection. Furthermore, we develop an evaluation framework based on the VISUS dataset and well-known performance metrics from the field of activity recognition. We systematically evaluate our methods within this framework, discuss potentials and limitations, and propose ways to improve the performance of future automatic visual attention detection methods.

Where and What: Driver Attention-based Object Detection

“…The evaluation between different grids is shown in Tab. 4. "Pixel-level" refers to the evaluation of the saliency map using 𝐷 𝐾𝐿 and 𝐶𝐶 metrics.…”

“…Previous works [20,53] set out to reduce tedious labelling by using gazeobject mapping, which annotates objects at the fixation level, i.e., the object being looked at. One popular algorithm checks whether a fixation lies in the object bounding box predicted by deep neural network-based object detector [4,21,29] such as YOLOv4 [5]. Wolf et al [53] suggest to use object segmentation using Mask-RCNN [12] as object area detection.…”

“…This algorithm from [3] can be used in Augmented Reality settings for cognition-aware mobile user interaction. In the follow-up work [4], the authors compare the mapping algorithms based on image cropping (IC) with object detectors (OD) in metrics such as precision and recall, and the results show that IC achieves higher precision but lower recall scores compared to OD.…”

“…Many computer vision applications embrace human gaze information, for instance in classification tasks [28,41], computer-aided medical diagnosis systems [16,42], or important objects selection/cropping in images and videos [43,44,50,52]. To better understand how the human brain processes visual stimuli, knowing not only where humans are looking at, but also what object is essential, i.e., gaze-object mapping [4]. This mapping is needed in many research projects, especially in analytics of student learning process [21] or human cognitive functions [35].…”

VIS-iTrack: Visual Intention Through Gaze Tracking Using Low-Cost Webcam

“…Graphical information includes discrete points of focus, whereas; textual information includes continuous points of focus. Based on prior studies [8], [12], [27], [28], [45], we considered fixation duration of 200ms as a threshold to define a fixation event. Intuitively, the fixation counts also increase proportionally with fixation duration.…”