Self-Supervised Spatio-Temporal Representation Learning for Videos by Predicting Motion and Appearance Statistics

Abstract

We address the problem of video representation learning without human-annotated labels. While previous efforts address the problem by designing novel self-supervised tasks using video data, the learned features are merely on a frame-by-frame basis, which are not applicable to many video analytic tasks where spatio-temporal features are prevailing. In this paper we propose a novel self-supervised approach to learn spatio-temporal features for video representation. Inspired by the success of two-stream approaches in video classification, we propose to learn visual features by regressing both motion and appearance statistics along spatial and temporal dimensions, given only the input video data. Specifically, we extract statistical concepts (fastmotion region and the corresponding dominant direction, spatio-temporal color diversity, dominant color, etc.) from simple patterns in both spatial and temporal domains. Unlike prior puzzles that are even hard for humans to solve, the proposed approach is consistent with human inherent visual habits and therefore easy to answer. We conduct extensive experiments with C3D to validate the effectiveness of our proposed approach. The experiments show that our approach can significantly improve the performance of C3D when applied to video classification tasks. Code is available at https://github.com/laura-wang/video repres mas.

Mutual information guided 3D ResNet for self‐supervised video representation learning

“…All the approaches employ RGB frames as input. Particularly, the Max‐GL is compared with DrLim [44], TempoCoh [11], object patch [9], temporal order [6], Odd‐One‐Out (O3N) [7], Order Prediction Network (OPN) [8], Generative Adversarial Network for Video (VGAN) [30], geometry [28], motion pattern [45], and video jigsaw [46]. As shown in Table 4, our Max‐GL achieves 64.0% on the UCF101 dataset, which improves 3.7% than the second best approach reported in [7].…”

Object Priors for Classifying and Localizing Unseen Actions

“…The general pipeline is to train a pre-text task on unlabeled data and transfer the knowledge to a supervised downstream task (Jing and Tian 2020) or by clustering video datasets without manual supervision (Asano et al 2020). Pretext tasks include dense predictive coding (Han et al 2020), shuffling frames (Fernando et al 2017;Xu et al 2019), exploiting spatial and/or temporal order (Jenni et al 2020;Tschannen et al 2020;Wang et al 2019), or by matching frames with other modalities (Afouras et al 2020;Alayrac et al 2020;Owens and Efros 2018;Patrick et al 2020). Self-supervised approaches utilize unlabeled train videos to learn representations without semantic class labels.…”

Consistent constraint-based video-level learning for action recognition

“…In addition, some works applied the self-supervised approaches to learn video feature based on multi-stream structure. Wang et al [23] proposed a two-stream-based self-supervised approach to learn visual feature by regressing both motion and appearance statistical information without action label. In this work, both RGB data and optical data were used to compute appearance and motion respectively.…”

“…2. Referring to the clip-level learning methods [1,2,23], the length of the clip is set to 16 frames in this paper.…”

“…In order to make the parameters of network as few as possible, we only use 5 convolution layers and 5 pooling layers (each convolution layer is immediately followed by a pooling layer), 2 fully connected layers and a softmax layer to predict action labels. Inspired by the previous works of 3D ConvNets [1,2,23], all of the convolutional kernels are set to 3 * 3 * 3 in the proposed approach. To improve the action recognition performance of 3D ConvNets, the video-level learning strategy is proposed in this paper.…”