Attention Prediction in Egocentric Video Using Motion and Visual Saliency

Abstract

We propose a method of predicting human egocentric visual attention using bottom-up visual saliency and egomotion information. Computational models of visual saliency are often employed to predict human attention; however, its mechanism and effectiveness have not been fully explored in egocentric vision. The purpose of our framework is to compute attention maps from an egocentric video that can be used to infer a person's visual attention. In addition to a standard visual saliency model, two kinds of attention maps are computed based on a camera's rotation velocity and direction of movement. These rotation-based and translation-based attention maps are aggregated with a bottom-up saliency map to enhance the accuracy with which the person's gaze positions can be predicted. The efficiency of the proposed framework was examined in real environments by using a head-mounted gaze tracker, and we found that the egomotion-based attention maps contributed to accurately predicting human visual attention.

An Introduction to the 3rd Workshop on Egocentric (First-Person) Vision

“…To date researchers have explored the use of egocentric vision for activity recognition [60,11,10], object recognition [63,14], summarization [31,35], temporal segmentation [68,26,61], scene understanding [65], interaction analysis [13,66], hand detection [33,32], gaze estimation [76], gaze analysis [54,80,34,10], visual saliency [78,79], social saliency [57] and motion capture [67].…”

“…Sousveillance ("undersight") refers to the less hierarchical and more rhizomic veillance of social networking, distributed cloud-based computing, self-sensing, bodyworn vision systems, wearable cameras [48,44,45,77,15,52,22,4], ego-centric vision (i.e. Personal Imaging) [40,37,58,38,31,12,79], implantable vision (firstperson point-of-eye sensing), and Mobile Pervasive Sensing [6]. Sousveillance with social networking is also an important area of study regarding privacy, security, and trust [19].…”

Computational Models of Human Visual Attention and Their Implementations: A Survey

“…† Overall saliency maps computed using the process (class (b)) are called "activation map" in the guided search model [6], [80], [81] or "priority map" in cognitive science [82]. [96], [97] Esaliency [63], Ozeki et al [98] Some typical computational models are classified according to the above two categories as shown in Table 1. To the best of our knowledge there are no models in the two blank cells.…”

“…Yamada et al focused on human egocentric visual attention during walking [96], [97]. They generated an egomotion-based attention map by integrating motion maps using egomotion information and a purely bottomup saliency map.…”

“…The computation consists of the following steps: 1) estimating camera motion (rotation and translation) from an egocentric video that includes visual motions caused by human head motions, 2) computing angular velocity and generating a rotation-based attention map, 3) computing the focus of expansion (FOE) of a moving scene and generating a translation-based attention map, 4) combining the maps and the graph-based visual saliency [106] using equal weights. They demonstrated that the combination of the bottom-up saliency map and the rotation-based attention map could achieve the most accurate predictions of human attention in egocentric scenes [97], whereas they revealed that saliency maps using typical dynamic features (motion and flicker) reduced a prediction accuracy [96].…”