A Real-Time Action Representation With Temporal Encoding and Deep Compression

Abstract

Deep neural networks have achieved remarkable success for video-based action recognition. However, most of existing approaches cannot be deployed in practice due to the high computational cost. To address this challenge, we propose a new real-time convolutional architecture, called Temporal Convolutional 3D Network (T-C3D), for action representation. T-C3D learns video action representations in a hierarchical multi-granularity manner while obtaining a high process speed. Specifically, we propose a residual 3D Convolutional Neural Network (CNN) to capture complementary information on the appearance of a single frame and the motion between consecutive frames. Based on this CNN, we develop a new temporal encoding method to explore the temporal dynamics of the whole video. Furthermore, we integrate deep compression techniques with T-C3D to further accelerate the deployment of models via reducing the size of the model. By these means, heavy calculations can be avoided when doing the inference, which enables the method to deal with videos beyond real-time speed while keeping promising performance. Our method achieves clear improvements on UCF101 action recognition benchmark against state-of-the-art real-time methods by 5.4% in terms of accuracy and 2 times faster in terms of inference speed with a less than 5MB storage model. We validate our approach by studying its action representation performance on four different benchmarks over three different tasks. Extensive experiments demonstrate comparable recognition performance to the stateof-the-art methods. The source code and the pre-trained models are publicly available at https://github.com/tc3d.

[Retracted] CNN‐LSTM Hybrid Model for Kinematic Feature Analysis and Parabolic Radian Prediction in Basketball Videos

“…e study in [34] found that video streaming data has a spatiotemporal correlation feature, which makes the video streams between connected regions influence each other. While increasing the spatiotemporal feature extraction of video stream sequences, this paper proposes to enhance the robustness and convergence of the CNN-LSTM prediction model using the RAdam optimization algorithm [24], which reduces the prediction error of the proposed model while reducing the model training time, and finally obtains a good prediction accuracy [35].…”

Efficient anomaly recognition using surveillance videos

“…They have worked through enriching a considerable portion of an existing dataset with spatial and temporal domains with annotations, using spatiotemporal tubes instead of whole-frame video segments. Liu et al (2020) have performed real time action representation using temporal encoding along with deep compression network. Their framework extracts video representations in a hierarchical granularity manner and use residual 3D CNN to extract appearance-based data.…”

Action Recognition Using Multi-Scale Temporal Shift Module and Temporal Feature Difference Extraction Based on 2D CNN

“…T-C3D, a framework proposed by Liu et al [26], first divides the given input video into three clips and samples eight frames from each clip. Thus, it can capture short-term features from frames in the same clips using 3D kernels and long-term features when fusing the prediction from each clip.…”