Spatiotemporal Pyramid Network for Video Action Recognition

Abstract

Two-stream convolutional networks have shown strong performance in video action recognition tasks. The key idea is to learn spatiotemporal features by fusing convolutional networks spatially and temporally. However, it remains unclear how to model the correlations between the spatial and temporal structures at multiple abstraction levels. First, the spatial stream tends to fail if two videos share similar backgrounds. Second, the temporal stream may be fooled if two actions resemble in short snippets, though appear to be distinct in the long term. We propose a novel spatiotemporal pyramid network to fuse the spatial and temporal features in a pyramid structure such that they can reinforce each other. From the architecture perspective, our network constitutes hierarchical fusion strategies which can be trained as a whole using a unified spatiotemporal loss. A series of ablation experiments support the importance of each fusion strategy. From the technical perspective, we introduce the spatiotemporal compact bilinear operator into video analysis tasks. This operator enables efficient training of bilinear fusion operations which can capture full interactions between the spatial and temporal features. Our final network achieves state-of-the-art results on standard video datasets.

Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization

“…Recently, researchers have shown increasing interests in exploring structured layers to enhance representation capability of networks [12,25,1,22]. One particular kind of structured layer is concerned with global covariance pooling after the last convolution layer, which has shown impressive improvement over the classical firstorder pooling, successfully used in FGVC [25], visual question answering [15] and video action recognition [34]. Very recent works have demonstrated that matrix square root normalization of global covariance pooling plays a key role in achieving state-of-the-art performance in both large-scale visual recognition [21] and challenging FGVC [24,32].…”

STM: SpatioTemporal and Motion Encoding for Action Recognition

“…However, these methods only model the appearance feature of each frame independently while ignore the dynamics between frames, which results in inferior performance when recognizing temporal-related videos. To handle the mentioned drawback, two-stream based methods [10,33,36,3,9] are introduced by modeling appearance and dynamics separately with two networks and fuse two streams through middle or at last. Among these methods, Simonyan et al [22] first proposed the two-stream ConvNet architecture with both spatial and temporal networks.…”

“…The existing methods for action recognition can be summarized into two categories. The first type is based on twostream neural networks [10,33,36,9], which consists of an RGB stream with RGB frames as input and a flow stream with optical flow as input. The spatial stream models the appearance features (not spatiotemporal features) without considering the temporal information.…”

Modality Compensation Network: Cross-Modal Adaptation for Action Recognition

“…With the advent of deep learning, neural networks are recently employed for action recognition due to its powerful ability in learning robust feature representations [3]- [6], [20]- [27]. The two-stream architecture [3] is a pioneer work to employ deep convolutional network for action recognition in videos, and has become a backbone of many other approaches [4]- [6], [25], [27]- [29]. To address the aggregation of spatial and temporal features, [6] explored different score fusion schemes.…”