Temporal Bilinear Networks for Video Action Recognition

Li, Yanghao; Song, Sijie; Li, Yuqi; Liu, Jiaying

doi:10.1609/aaai.v33i01.33018674

Cited by 41 publications

(29 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…From Fig. 6 we have the following findings for RGB frames: 1) Basic* fails over BESS: 13,19,22,23,24,34,38,42,43,45,46 2,8,12,13,18,19,21,22,25,28,33,34,36,38,40,42,43,44,46,47,51 From Fig. 7 we have the following findings for Flow frames: 1) Basic* fails over BESS: 8,12,17,19,20,24,34,36,38,41,43,45,46 2,5,6,9,…”

Section: Effectiveness Of Bess and Hafsmentioning

confidence: 97%

“…In recent years, action recognition performance in videos has improved with large margin through two-stream network architecture [7]. Inspired by the work in [7], numerous approaches have been proposed considering two-stream architecture as the backbone structure [8] - [10], [23] [26] [28]. In two stream network, one stream is dedicated for RGB and another one for flow frames to extract appearance and temporal features respectively.…”

Section: Related Work a Rgb And Flow Based Action Recognitionmentioning

confidence: 99%

See 1 more Smart Citation

Batch Entropy Supervised Convolutional Neural Networks for Feature Extraction and Harmonizing for Action Recognition

et al. 2020

View full text Add to dashboard Cite

Section: Effectiveness Of Bess and Hafsmentioning

confidence: 97%

Section: Related Work a Rgb And Flow Based Action Recognitionmentioning

confidence: 99%

Batch Entropy Supervised Convolutional Neural Networks for Feature Extraction and Harmonizing for Action Recognition

et al. 2020

View full text Add to dashboard Cite

“…In the past decades, Deep Neural Network (DNN), as an effective data-driven solution for computer vision tasks, has been exploited to accomplish high-level visual recognition tasks, e.g. image classification [12], action recognition [13,14], and low-level image restoration/enhancement tasks, including super-resolution [15,16,17], low-light enhancement [18,19], rain removal [20,21,22,23], etc. With powerful computing devices like GPUs and TPUs [24], given well-defined inputs and outputs, the network can automatically learn an end-to-end mapping from inputs to outputs.…”

Section: Introductionmentioning

confidence: 99%

Progressive Spatial Recurrent Neural Network for Intra Prediction

Yang

et al. 2019

IEEE Trans. Multimedia

Self Cite

View full text Add to dashboard Cite

Intra prediction is an important component of modern video codecs, which is able to efficiently squeeze out the spatial redundancy in video frames. With preceding pixels as the context, traditional intra prediction schemes generate linear predictions based on several predefined directions (i.e. modes) for blocks to be encoded. However, these modes are relatively simple and their predictions may fail when facing blocks with complex textures, which leads to additional bits encoding the residue. In this paper, we design a Progressive Spatial Recurrent Neural Network (PS-RNN) that learns to conduct intra prediction. Specifically, our PS-RNN consists of three spatial recurrent units and progressively generates predictions by passing information along from preceding contents to blocks to be encoded. To make our network generate predictions considering both distortion and bit-rate, we propose to use Sum of Absolute Transformed Difference (SATD) as the loss function to train PS-RNN since SATD is able to measure rate-distortion cost of encoding a residue block. Moreover, our method supports variable-block-size for intra prediction, which is more practical in real coding conditions. The proposed intra prediction scheme achieves on average 2.5% bit-rate reduction on variable-block-size settings under the same reconstruction quality compared with HEVC.

show abstract

“…Yet our approach does not utilize optical flow or the two-stream structure during both training and inference, leading to a significant reduction of more than 83.2% and 28.8% in parameters and FLOPs, respectively. For Something-Something v1, while our MF-KPSEM performs slightly inferior compared to S3D [83] and TSM [92], our MF-KPSEM is a much lighter network with smaller parameter size and requires less computational cost. Specifically, the parameters of our MF-KPSEM are…”

Section: Results and Comparisonmentioning

confidence: 89%

“…• CNN with learnable feature correlations: TBN [92], Res50-NL [8], Res50-CGD [93], Res50-CGNL [94], I3D-NL [8] and I3D-NL-GCN [95]. The performance results in Table 3.1 show that our network achieves the state-ofthe-art result on the Mini-Kinetics with only a minor increase in the number of parameters and required computational cost.…”

Section: Results and Comparisonmentioning

confidence: 98%

Effective action recognition with fully supervised and self-supervised methods

Cao¹

View full text Add to dashboard Cite

Action recognition in videos has attracted interest in computer vision and machine learning communities thanks to its applications such as surveillance and smart homes. In addition to spatial information in individual frames, videos contain temporal information across the temporal dimension. Therefore, effective spatiotemporal representation is the key to accurate action recognition in videos.Previous works have proposed various fully-supervised and self-supervised methods for video representation learning. For fully-supervised methods, most of them utilize convolution neural networks (CNNs) to extract spatial representation while temporal representation is usually modelled by pixel-wise correlations. However, it is inefficient to extract correlations between all pixels since some of them may relate to in-salient area (e.g. backgrounds or environments). On the other hand, self-supervised methods are proposed to leverage more accessible unlabled data on the Internet and transfer the extracted representation for different downstream tasks. The core of self-supervised methods is to design a pretext task where supervision signal is automatically generated based on characteristics of unlabeled data.Although self-supervised methods avoid the annotation of labeled data, compared to fully-supervised methods, there is room for performance improvement of selfsupervised methods. In this thesis, we address the above research gap with two novel deep learning methods, to advance fully-supervised and self-supervised methods, respectively. For fully-supervised learning, we propose a novel Key Point Shift Embedding Module (KPSEM) to adaptively extract channel-wise key point shifts across video frames without key point annotation for temporal feature extraction. Key points are adaptively extracted as feature points with maximum feature values at split regions, while key point shifts are the spatial displacements of corresponding key points. The key point shifts are encoded as the overall temporal features via linear embedding layers in a multi-set manner. vi To advance self-supervised learning, we propose a novel self-supervised learning method, called Video Incoherence Detection (VID), that leverages incoherence detection for spatio-temporal feature extraction. It roots from the observation that visual systems of human beings can easily identify video incoherence based on their comprehensive understanding of videos. Specifically, the training sample, denoted as the incoherent clip, is constructed by multiple sub-clips hierarchically sampled from the same raw video with various lengths of incoherence between each other. The network is trained to learn high-level representation by predicting the relative location and length of incoherence given the incoherent clip as input. Additionally, intra-video contrastive learning is introduced to maximize the mutual information between different incoherent clips from the same raw video. Our experiments show that both KPSEM and VID achieve state-of-the-art performance on action recognition wi...

show abstract

Temporal Bilinear Networks for Video Action Recognition

Cited by 41 publications

References 27 publications

Batch Entropy Supervised Convolutional Neural Networks for Feature Extraction and Harmonizing for Action Recognition

Batch Entropy Supervised Convolutional Neural Networks for Feature Extraction and Harmonizing for Action Recognition

Progressive Spatial Recurrent Neural Network for Intra Prediction

Effective action recognition with fully supervised and self-supervised methods

Contact Info

Product

Resources

About