Gate-Shift Networks for Video Action Recognition

Abstract

Most action recognition methods base on a) a late aggregation of frame level CNN features using average pooling, max pooling, or RNN, among others, or b) spatio-temporal aggregation via 3D convolutions. The first assume independence among frame features up to a certain level of abstraction and then perform higher-level aggregation, while the second extracts spatiotemporal features from grouped frames as early fusion. In this paper we explore the space in between these two, by letting adjacent feature branches interact as they develop into the higher level representation. The interaction happens between feature differencing and averaging at each level of the hierarchy, and it has convolutional structure that learns to select the appropriate mode locally in contrast to previous works that impose one of the modes globally (e.g. feature differencing) as a design choice. We further constrain this interaction to be conservative, e.g. a local feature subtraction in one branch is compensated by the addition on another, such that the total feature flow is preserved. We evaluate the performance of our proposal on a number of existing models, i.e. TSN, TRN and ECO, to show its flexibility and effectiveness in improving action recognition performance.

Metric-Based Attention Feature Learning for Video Action Recognition

“…Table V analyzes the generalization ability of the proposed method for the Something-V1 dataset. Note that the proposed method achieved SOTA performance of 52.08%, surpassing the latest methods [18,35] that require a vast amount of computation over 50 GFLOPs. In addition, as the number of frames used for learning decreased, the proposed method had a higher performance than GSN.…”

“…As the training and evaluation datasets changed, a different backbone structure was used in this experiment. In detail, the InceptionV3 of GSN [35] was employed as a backbone, and the classifier was trained after attaching the DA module with the output terminal of InceptionV3 (see Fig. 2).…”

“…Sudhakaran et al proposed a few gating methods to upgrade the spatio-temporal decomposition of 3D convolution kernel, and achieved the SOTA performance [35]. Our approach differs from the gate-shift network (GSN) of [35] in two respects: 1) Unlike GSN, which considers only the local view, the proposed method analyzes the image characteristics from a global perspective, similar to the nonlocal network [37]. 2) Through discriminative learning based on feature map similarity, the proposed method can emphasize the target object(s).…”

“…In this paper, ResNext-101 [1] was used as the backbone for UCF-101 and HMDB-51 datasets, and InceptionV3 of GSN [35] was adopted as the backbone for Diving48 and Something-V1 datasets. To construct the same experimental environment as the previous techniques, pre-trained ResNext-101 with different datasets and InceptionV3 architectures were used in the fine-tuning process of action recognition.…”

Learning to Recognize Actions on Objects in Egocentric Video With Attention Dictionaries

“…Another alternative is to extract appearance features from the individual frames and perform a temporal pooling operation to encode their temporal evolution [10], [13], [68]. Recent approaches explore the feasibility of temporal modeling with 2D CNNs [34], [55], [65]. Another approach includes using two CNNs, each encoding an RGB image for appearance cues and stacks of optical flow for motion cues [11], [12], [51].…”