SAST: Learning Semantic Action-Aware Spatial-Temporal Features for Efficient Action Recognition

Abstract

The state-of-the-arts in action recognition are suffering from three challenges: (1) How to model spatial transformations of action since it is always geometric variation over time in videos. (2) How to develop the semantic action-aware temporal features from one video with a large proportion of irrelevant frames to the labeled action class, which hurt the final performance. (3) The action recognition speed of most existing models is too slow to be applied to actual scenes. In this paper, to address these three challenges, we propose a novel CNN-based action recognition method called SAST including three important modules, which can effectively learn semantic action-aware spatial-temporal features with a faster speed. Firstly, to learn action-aware spatial features (spatial transformations), we design a weight shared 2D Deformable Convolutional network named 2DDC with deformable convolutions whose receptive fields can be adaptively adjusted according to the complex geometric structure of actions. Then, we propose a light Temporal Attention model called TA to develop the action-aware temporal features that are discriminative for the labeled action category. Finally, we apply an effective 3D network to learn the temporal context between frames for building the final video-level representation. To improve the efficiency, we only utilize the raw RGB rather than optical flow and RGB as the input to our model. Experimental results on four challenging video recognition datasets Kinetics-400, Something-Something-V1, UCF101 and HMDB51 demonstrate that our proposed method can not only achieve comparable performances but be 10x to 50x faster than most of state-of-the-art action recognition methods.

Vision-based Human Activity Recognition Using Local Phase Quantization

“…It results more computational time compared to single feature for extracting feature as well as classifying activity. On the other hand, the proposed approach is outperforming the various DL method [36,37,38,39,59,60,61] that shown in the Table 2. The major challenge with DL methods is that it requires large sampled data to classify actions efficiently.…”

“…They have achieved 93.6% and 66.2% accuracies on UCF101 and HMDB51 datasets, respectively. Wang et al [39] have proposed CNN based Semantic Action-Aware Spatial-Temporal Features for action recognition with 71.2%, 45.6%, 95.9%, and 74.8% accuracies on Kinetics-400, Something-Something-V1, UCF101 and HMDB51 datasets, respectively. Xia and Wen [59] proposed a multi-stream based on key frame sampling for HAR.…”

An Improved Action Recognition Network With Temporal Extraction and Feature Enhancement

“…TSN [9] is the baseline, followed by some 3D CNN based methods, including I3D [18], ECO [22] and SAST [48] in the middle, and some 2D CNN based methods, including TRN [17], TPN [2], TSM [15] MTD 2 P [49] and CorrNet [12] and the lower part.Compare the results of 8 frames,our network exceeds other 3D based methods on V1. As for the results of 24 frames, we surpass ECOEn [22] and SAST [48] by 5% and 5.8%, respectively. In the meantime, those 3D based methods are pre-trained on the very large dataset (e.g.…”

Spatial-Temporal Information Aggregation and Cross-Modality Interactive Learning for RGB-D-Based Human Action Recognition

“…Since the 2D CNN cannot explicitly learn the temporal information in the video, some works adopted sparse temporal sampling strategies [5], [22], [23] or explored temporal dependencies between video frames at multiple time scales [24]- [26]. Compared with 2D CNN, 3D CNN-based methods [27]- [30] are able to directly extract spatial-temporal features from videos. Although these works have achieved good performance, the use of 3D CNN is limited by its parameters and computational overhead, prompting the emergence of the works on 3D convolutional kernel factorization [31]- [33] to balance the accuracy and model cost.…”