Attend and Interact: Higher-Order Object Interactions for Video Understanding

Abstract

Human actions often involve complex interactions across several inter-related objects in the scene. However, existing approaches to fine-grained video understanding or visual relationship detection often rely on single object representation or pairwise object relationships. Furthermore, learning interactions across multiple objects in hundreds of frames for video is computationally infeasible and performance may suffer since a large combinatorial space has to be modeled. In this paper, we propose to efficiently learn higher-order interactions between arbitrary subgroups of objects for fine-grained video understanding. We demonstrate that modeling object interactions significantly improves accuracy for both action recognition and video captioning, while saving more than 3-times the computation over traditional pairwise relationships. The proposed method is validated on two large-scale datasets: Kinetics and ActivityNet Captions. Our SINet and SINet-Caption achieve state-of-the-art performances on both datasets even though the videos are sampled at a maximum of 1 FPS. To the best of our knowledge, this is the first work modeling object interactions on open domain large-scale video datasets, and we additionally model higher-order object interactions which improves the performance with low computational costs. * Work performed as a NEC Labs intern Attended interactions Interactions between ROIs Video frame Action prediction: cooking on campfire , cooking egg , …

Temporal Attentive Alignment for Large-Scale Video Domain Adaptation

“…Another branch of work utilizes optical flow to compensate for the lack of temporal information in raw RGB frames [42,9,49,3,29]. Moreover, some works extract temporal dependencies between frames for video tasks by utilizing recurrent neural networks (RNNs) [6], attention [28,30] and relation modules [57]. Note that we focus on attending to the temporal dynamics to effectively align domains and we consider other modalities, e.g.…”

Grounded Video Description

“…Visual attention usually comes in the form of temporal attention [35] (or spatial-attention [33] in the image domain), semantic attention [14,36,37,42] or both [20]. The recent unprecedented success in object detection [24,7] has regained the community's interests on detecting fine-grained visual clues while incorporating them into end-toend networks [17,27,1,16]. Description methods which are based on object detectors [17,39,1,16,5,13] tackle the captioning problem in two stages.…”

“…The recent unprecedented success in object detection [24,7] has regained the community's interests on detecting fine-grained visual clues while incorporating them into end-toend networks [17,27,1,16]. Description methods which are based on object detectors [17,39,1,16,5,13] tackle the captioning problem in two stages. They first use offthe-shelf or fine-tuned object detectors to propose object proposals/detections as for the visual recognition heavylifting.…”

“…They first use offthe-shelf or fine-tuned object detectors to propose object proposals/detections as for the visual recognition heavylifting. Then, in the second stage, they either attend to the object regions dynamically [17,39,1] or classify the regions into labels and fill into pre-defined/generated sentence templates [16,5,13]. However, directly generating proposals from off-the-shelf detectors causes the proposals to bias towards classes in the source dataset (i.e.…”

A Structured Model for Action Detection

“…Visual relationship modeling for human-human and human-object pairs increases performance in a variety of tasks including action recognition [56] and image captioning [35,38]. There have been several works [5,13,16] on human-object interaction modeling in images that achieved significant improvements on HICO-DET [6] and V-COCO [32] datasets.…”