Time and Frequency Network for Human Action Detection in Videos

Abstract

Currently, spatiotemporal features are embraced by most deep learning approaches for human action detection in videos, however, they neglect the important features in frequency domain. In this work, we propose an end-to-end network that considers the time and frequency features simultaneously, named TFNet. TFNet holds two branches, one is time branch formed of three-dimensional convolutional neural network(3D-CNN), which takes the image sequence as input to extract time features; and the other is frequency branch, extracting frequency features through two-dimensional convolutional neural network(2D-CNN) from DCT coefficients. Finally, to obtain the action patterns, these two features are deeply fused under the attention mechanism. Experimental results on the JHMDB51-21 and UCF101-24 datasets demonstrate that our approach achieves remarkable performance for frame-mAP.

Explore Spatio-temporal Aggregation for Insubstantial Object Detection: Benchmark Dataset and Baseline

“…The frame-level detectors [56,69] generate final action tubes by linking frame level detection results. To take full advantage of temporal information, some clip-level detectors have been proposed [24,40,44,45,51,55,60,66,74,79]. ACT [40] handled a short sequence of frames and output action tubes by regressing from anchor cuboids.…”

“…Sarmiento et al [60] introduced two cross attention blocks to effectively model the spatial relations and capture short range temporal interactions. In recent years, some approaches attempt to recognize action based on 3D convolution features [32,44,51,55,79]. ACDnet [51] intelligently exploited the temporal coherence between successive video frames to approximate their CNN features rather than naively extracting them.…”

“…ACDnet [51] intelligently exploited the temporal coherence between successive video frames to approximate their CNN features rather than naively extracting them. TFNet [44] applied attention mechanism to fuse the temporal features extracted by 3D-CNN and the frequency features extracted by 2D-CNN. ACAR-Net [55] built upon a novel high-order relation reasoning operator and an actor-context feature bank to enable indirect relation reasoning.…”