Ego2Top: Matching Viewers in Egocentric and Top-View Videos

Abstract

Abstract. Egocentric cameras are becoming increasingly popular and provide us with large amounts of videos, captured from the first person perspective. At the same time, surveillance cameras and drones offer an abundance of visual information, often captured from top-view. Although these two sources of information have been separately studied in the past, they have not been collectively studied and related. Having a set of egocentric cameras and a top-view camera capturing the same area, we propose a framework to identify the egocentric viewers in the top-view video. We utilize two types of features for our assignment procedure. Unary features encode what a viewer (seen from top-view or recording an egocentric video) visually experiences over time. Pairwise features encode the relationship between the visual content of a pair of viewers. Modeling each view (egocentric or top) by a graph, the assignment process is formulated as spectral graph matching. Evaluating our method over a dataset of 50 top-view and 188 egocentric videos taken in different scenarios demonstrates the efficiency of the proposed approach in assigning egocentric viewers to identities present in top-view camera. We also study the effect of different parameters such as the number of egocentric viewers and visual features. An extended abstract of this effort has been reported in [1].

Multi-Channel Attention Selection GAN With Cascaded Semantic Guidance for Cross-View Image Translation

“…We also provide the visualization results of the generated uncertainty maps (Sec. 8) and the arbitrary cross-view image translation experiments on Ego2Top dataset [1] (Sec. 9).…”

“…Dayton [42] and CVUSA [44]. Meanwhile, we also create a largerscale cross-view synthesis benchmark using the data from Ego2Top [1], and present results of multiple baseline models for the research community.…”

Egocentric Vision-based Future Vehicle Localization for Intelligent Driving Assistance Systems

“…An egocentric camera view is often the most natural perspective for observing an ego-vehicle environment, but it introduces additional challenges due to its narrow field of view. The literature in egocentric visual perception has typically focused on activity recognition [5], [6], [11]- [13], object detection [14]- [16], person identification [17]- [19], video summarization [20], and gaze anticipation [21]. Recently, papers have also applied egocentric vision to ego-action estimation and prediction.…”

Joint Person Segmentation and Identification in Synchronized First- and Third-Person Videos

“…First-person Cameras. Ardeshir and Borji [4] match a set of first-person videos to a set of people appearing in a top-view video using graph matching, but assume there are multiple first-person cameras sharing the same field of view at any time and only consider third-person cameras that are overhead. Fan et al [14] identify a first-person camera wearer in a third-person video using a two-stream semi-Siamese network that incorporates spatial and temporal information from both views, and learns a joint embedding space from first-and third-person matches.…”

“…While person tracking and (re-)identification are well-studied in computer vision [37,44], only recently have they been considered in challenging scenarios of heterogeneous first-person and traditional cameras. Ardeshir and Borji [4] consider the case of several people moving around while wearing cameras, and try to match each of these first-person views to one of the people appearing in a third-person, overhead view of the scene. This is challenging because the camera wearer is never seen in their own wearable video, so he or she must be identified by matching their motion from a third-person perspective with the first-person visual changes that are induced by their movements.…”