Spatio-Temporal Action Graph Networks

Abstract

Events defined by the interaction of objects in a scene are often of critical importance; yet important events may have insufficient labeled examples to train a conventional deep model to generalize to future object appearance. Activity recognition models that represent object interactions explicitly have the potential to learn in a more efficient manner than those that represent scenes with global descriptors. We propose a novel inter-object graph representation for activity recognition based on a disentangled graph embedding with direct observation of edge appearance. In contrast to prior efforts, our approach uses explicit appearance for high order relations derived from objectobject interaction, formed over regions that are the union of the spatial extent of the constituent objects. We employ a novel factored embedding of the graph structure, disentangling a representation hierarchy formed over spatial dimensions from that found over temporal variation. We demonstrate the effectiveness of our model on the Charades activity recognition benchmark, as well as a new dataset of driving activities focusing on multi-object interactions with near-collision events. Our model offers significantly improved performance compared to baseline approaches without object-graph representations, or with previous graphbased models.

Video Action Transformer Network

“…Attention for action recognition: There has been a large body of work on incorporating attention in neural networks, primarily focused on language related tasks [44,51]. Attention for videos has been pursued in various forms, including gating or second order pooling [12,30,31,49], guided by human pose or other primitives [4,5,12,13], regiongraph representations [19,48], recurrent models [37] and self-attention [47]. Our model can be thought of as a form of self-attention complementary to these approaches.…”

Smart Black Box 2.0: Efficient High-Bandwidth Driving Data Collection Based on Video Anomalies

“…Other approaches use physiological signals from the driver [22]. In recent years, deep-learning computer vision techniques have been applied to anomaly detection in first-person driving videos [14,15,23,24]. Other works further attempt to classify the type of anomaly occurring in the video, either offline after the video is fully observed [25][26][27] or in real time [16].…”

“…STAG [24] Anomaly detection (supervised) Uses a spatio-temporal action graph (STAG) network to model the spatial and temporal relations among objects.…”

Language-Conditioned Graph Networks for Relational Reasoning

“…The core idea is to enable communication between image regions to build contextualized representations of these regions. Graph networks have been successfully applied to various tasks, from object detection [25] and region classification [7] to human-object interaction [30] and activity recognition [12]. Besides, self-attention models [35] and non-local networks [38] can also be cast as graph networks in a general sense.…”