Object Detection-Based Location and Activity Classification from Egocentric Videos: A Systematic Analysis

Abstract

Egocentric vision has emerged in the daily practice of application domains such as lifelogging, activity monitoring, robot navigation and the analysis of social interactions. Plenty of research focuses on location detection and activity recognition, with applications in the area of Ambient Assisted Living. The basis of this work is the idea that indoor locations and daily activities can be characterized by the presence of specific objects. Objects can be obtained either from laborious human annotations or automatically, using vision-based detectors. We perform a study regarding the use of object detections as input for location and activity classification and analyze the influence of various detection parameters. We compare our detections against manually provided object labels and show that location classification is affected by detection quality and quantity. Utilization of the temporal structure in object detections mitigates the consequences of noisy ones. Moreover, we determine that the recognition of activities is related to the presence of specific objects and that the lack of explicit associations between certain activities and objects hurts classification performance for these activities. Finally, we discuss the outcomes of each task and our method's potential for real-world applications.

Multi-Dataset, Multitask Learning of Egocentric Vision Tasks

“…In [61], [62] optical flow was employed to detect salient regions, which were cropped from the original RGB frames and were given to the network as a second, more focused RGB stream. Other input modalities have been employed including depth [7], [41], egocentric cues comprising hand [63], [64], [65] and object regions [64], [66], [67], head motions [63] and gaze-based saliency maps [63], [65], sensor-based modalities [15], [56], [59] and sound [43], [68], [69]. In [38], [40] object and hand localization and segmentation were intermediate learning steps that forced the network to focus on important egocentric cues prior to action prediction.…”

A Modular Approach for the Detection and Interconnection of Objects, Hands, Locations, and Actions for Egocentric Video Understanding

“…• The Binary Presence Vector (BPV) of objects [68,70] from Chapter 3.2.3 consisting of zeros and ones with length equal to the number of noun classes of EPIC-Kitchens (352). The BPVs are concatenated to the hand coordinates for every frame and the feature size increases to 356 (352 + 4).…”

“…Furthermore, we reflect on the effect of object detection quality on the location and activity recognition outputs. Parts of this chapter are published in [68,70].…”

“…A number of distinct input modalities have been employed to assist in egocentric action recognition. These include depth [7,151], egocentric cues comprising hand [66,129,134] and object regions [66,68,164], head motions [134] and gaze-based saliency maps [129,134], sensor-based modalities [101,114,139] and sound [18,74,167]. Typically, these methods require specialized sensors such as depth cameras, eye trackers, accelerometers, or inertial measurement units for the additional inputs, whereas sound is provided from the built-in microphones of the camera.…”

“…Naturally, this refers to the surrounding area and its contents, usually consisting of objects, hands, other people, and the scene background. Being able to examine a perspective of the scene that accumulates this information with clarity allows for improved inference of higher level cues such as the quantification of interactions between hands based on their proximity [103], object-activity relations from associated movements [7], and location identification from the presence of distinctive objects [68,70] as shown in Chapter 3.…”