A Structured Model for Action Detection

Abstract

A dominant paradigm for learning-based approaches in computer vision is training generic models, such as ResNet for image recognition, or I3D for video understanding, on large datasets and allowing them to discover the optimal representation for the problem at hand. While this is an obviously attractive approach, it is not applicable in all scenarios. We claim that action detection is one such challenging problem -the models that need to be trained are large, and the labeled data is expensive to obtain. To address this limitation, we propose to incorporate domain knowledge into the structure of the model to simplify optimization. In particular, we augment a standard I3D network with a tracking module to aggregate long term motion patterns, and use a graph convolutional network to reason about interactions between actors and objects. Evaluated on the challenging AVA dataset, the proposed approach improves over the I3D baseline by 5.5% mAP and over the state-ofthe-art by 4.8% mAP.

Complete Video-Level Representations for Action Recognition

“…People and objects interacting with people should be the focus of activity recognition. Zhang et al [21] [28] object detection algorithms in deep learning show good performance. Overall, the two-stage object detection framework is more accurate, but the one-stage YOLO has faster, even real-time inference speeds with guaranteed accuracy, making it well suited for engineering practice.…”

Spot What Matters: Learning Context Using Graph Convolutional Networks for Weakly-Supervised Action Detection

“…While action recognition considers the classification task, action detection carries out both classification and detection of actions. Although action detection is usually addressed using full supervision [7,9,23,25,30,33], we are interested in weak supervision, which allows us to reduce annotation cost by training models using very few action bounding box annotations per action instance.…”

“…Recently, GCN have been used for visual relational reasoning for the tasks of action recognition [29] and group activity recognition [32]. Zhang et al [33] employ GCN [12] for action detection, where nodes represent detected actors and objects. As we do not require an external object detector, our approach is suitable to reason with respect to arbitrary context and objects which cannot be detected, e.g.…”

“…We are interested in an approach to learn these contextual cues using as few annotated data as possible. Recent works [7,23,25,33] that model contextual cues for action detection rely on full supervision in terms of actor bounding box annotations. However, extensive video annotation is time consuming and expensive [15].…”

Unsupervised co-segmentation of actions in motion capture data and videos

“…The recent surge of new methods in this context include various methods based on graph-based representations of videos and actions [87,161] and deep neural networks [8,82,219]. Most of those methods achieve impressive results for a series of challenging tasks, such as action detection [240,241], human-object interaction understanding [64,126,161,196,224] and relational reasoning in videos [8,244], action similarity assessment [162,215] and skill determination [52,53,120].…”