Learning Spatio-Temporal Representation With Local and Global Diffusion

Abstract

Convolutional Neural Networks (CNN) have been regarded as a powerful class of models for visual recognition problems. Nevertheless, the convolutional filters in these networks are local operations while ignoring the large-range dependency. Such drawback becomes even worse particularly for video recognition, since video is an information-intensive media with complex temporal variations. In this paper, we present a novel framework to boost the spatio-temporal representation learning by Local and Global Diffusion (LGD). Specifically, we construct a novel neural network architecture that learns the local and global representations in parallel. The architecture is composed of LGD blocks, where each block updates local and global features by modeling the diffusions between these two representations. Diffusions effectively interact two aspects of information, i.e., localized and holistic, for more powerful way of representation learning. Furthermore, a kernelized classifier is introduced to combine the representations from two aspects for video recognition. Our LGD networks achieve clear improvements on the large-scale Kinetics-400 and Kinetics-600 video classification datasets against the best competitors by 3.5% and 0.7%. We further examine the generalization of both the global and local representations produced by our pretrained LGD networks on four different benchmarks for video action recognition and spatio-temporal action detection tasks. Superior performances over several state-of-theart techniques on these benchmarks are reported. Code is available at: https://github.com/ZhaofanQiu/ local-and-global-diffusion-networks.

Enhanced Action Recognition Using Multiple Stream Deep Learning with Optical Flow and Weighted Sum

“…Table 6 shows comparison results with conventional methods on the UCF-101 dataset. LGD-3D Two-stream and PoTion + I3D showed similar accuracies to that of the proposed method, but the accuracy of the proposed method was higher on other datasets [ 6 , 36 ].…”

BiST: Bi-directional Spatio-Temporal Reasoning for Video-Grounded Dialogues

“…uses a shared network of 2D CNNs over three orthogonal views of video to obtain spatial and temporal signals for action recognition. (Qiu et al, 2019) adopts a twopath network architecture that integrates global and local information of both temporal and spatial dimensions for video classification. Other research areas that investigate spatio-temporal learning include video captioning (Aafaq et al, 2019), video super-resolution (Li et al, 2019b), and video object segmentation (Xu et al, 2019).…”

Decoupling Localization and Classification in Single Shot Temporal Action Detection

“…In=c log(pn), (6) where I n=c is an indicator function which equals to 1 if n is the ground truth class label c, otherwise 0. For location regression, we employ the Smooth L1 loss (S L1 ) to force the proposal (ϕ c , ϕ w ) to move towards its closest ground truth proposal (g c , g w ).…”

“…With the tremendous increase in online and personal media archives, people are generating, storing, and consuming a large collection of videos. This trend encourages the development of effective and efficient algorithms to intelligently parse video data [1,2,3,4,5,6] and discover semantic information [7,8]. One fundamental challenge underlying the success of these advances is action detection from videos in both temporal [9,10] and spatio-temporal aspects [11].…”