Gaussian Temporal Awareness Networks for Action Localization

Abstract

Temporally localizing actions in a video is a fundamental challenge in video understanding. Most existing approaches have often drawn inspiration from image object detection and extended the advances, e.g., SSD and Faster R-CNN, to produce temporal locations of an action in a 1D sequence. Nevertheless, the results can suffer from robustness problem due to the design of predetermined temporal scales, which overlooks the temporal structure of an action and limits the utility on detecting actions with complex variations. In this paper, we propose to address the problem by introducing Gaussian kernels to dynamically optimize temporal scale of each action proposal. Specifically, we present Gaussian Temporal Awareness Networks (GTAN)a new architecture that novelly integrates the exploitation of temporal structure into an one-stage action localization framework. Technically, GTAN models the temporal structure through learning a set of Gaussian kernels, each for a cell in the feature maps. Each Gaussian kernel corresponds to a particular interval of an action proposal and a mixture of Gaussian kernels could further characterize action proposals with various length. Moreover, the values in each Gaussian curve reflect the contextual contributions to the localization of an action proposal. Extensive experiments are conducted on both THUMOS14 and ActivityNet v1.3 datasets, and superior results are reported when comparing to state-of-the-art approaches. More remarkably, GTAN achieves 1.9% and 1.1% improvements in mAP on testing set of the two datasets.

A Survey on Temporal Action Localization

“…But SS-TAD adopts the anchor mechanism and the stacked GRU units. Recently, Fuchen Long et al introduces GTAN (Gaussian Temporal Awareness Networks) [25] which integrates temporal structure to one-stage action localization. In GTAN, it introduces Gaussian kernels to optimize temporal scale of every action proposal dynamically.…”

“…Furthermore, some region-based methods (such as R-C3D [23] and TAL-Net [24]) propose to generalize the methods for 2D object detection to 1D temporal action localization. Recently, TSA-Net [26] and Gaussian temporal modeling [25] are proposed for accurate action localization. The following are the performance comparisons.…”

Am I Done? Predicting Action Progress in Videos

“…Gkioxari et al generate action proposals by filtering Selective Search boxes with motion saliency, and fuse motion and temporal decision using an SVM [14]. More recent approaches, devised end-to-end tunable architectures integrating region proposal networks in their model [4,13,23,26,35,44,47,48,50,65]. As discussed in [47], most action detection works do not deal with untrimmed sequences and do not generate action tubes.…”

“…Action progress prediction is an extremely challenging task since, to be of maximum utility, the prediction should be made online while observing the video. While a thick crop of literature addresses action detection and spatio-temporal localization [12,13,23,26,35,44,59,62,66], predicting action progress is more closely related to online action detection [6,22,28,50,61]. Here the goal is to accurately detect, as soon as possible, when an action has started and when it has finished, but they do not have a model to estimate the progress.…”

Decoupling Localization and Classification in Single Shot Temporal Action Detection

“…This trend encourages the development of effective and efficient algorithms to intelligently parse video data [1,2,3,4,5,6] and discover semantic information [7,8]. One fundamental challenge underlying the success of these advances is action detection from videos in both temporal [9,10] and spatio-temporal aspects [11]. In this study, we focus on the temporal action detection task, which aims to find the exact time stamps of an action's start and end time, and recognize the category of the action.…”