Actions ~ Transformations

Long

2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

et al. 2017

239

117

Two-stream convolutional networks have shown strong performance in video action recognition tasks. The key idea is to learn spatiotemporal features by fusing convolutional networks spatially and temporally. However, it remains unclear how to model the correlations between the spatial and temporal structures at multiple abstraction levels. First, the spatial stream tends to fail if two videos share similar backgrounds. Second, the temporal stream may be fooled if two actions resemble in short snippets, though appear to be distinct in the long term. We propose a novel spatiotemporal pyramid network to fuse the spatial and temporal features in a pyramid structure such that they can reinforce each other. From the architecture perspective, our network constitutes hierarchical fusion strategies which can be trained as a whole using a unified spatiotemporal loss. A series of ablation experiments support the importance of each fusion strategy. From the technical perspective, we introduce the spatiotemporal compact bilinear operator into video analysis tasks. This operator enables efficient training of bilinear fusion operations which can capture full interactions between the spatial and temporal features. Our final network achieves state-of-the-art results on standard video datasets.

Section: Final Resultsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Spatiotemporal Pyramid Network for Video Action Recognition

Long

2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

et al. 2017

239

117

Lecture Notes in Computer Science

“…(c) Recurrent Spatial Networks [15] [22], which applies Recurrent Neural Networks, including LSTM or GRU to model temporal information in videos. (d) Other approaches [23] [24][25] [26], which use other solutions to generate compact features for video representation and classification.…”

Section: Deep Neural Network For Large-scale Video Classificationmentioning

confidence: 99%

NeXtVLAD: An Efficient Neural Network to Aggregate Frame-Level Features for Large-Scale Video Classification

Lin

Xiao

Fan

2019

This paper introduces a fast and efficient network architecture, NeXtVLAD, to aggregate frame-level features into a compact feature vector for large-scale video classification. Briefly speaking, the basic idea is to decompose a high-dimensional feature into a group of relatively low-dimensional vectors with attention before applying NetVLAD aggregation over time. This NeXtVLAD approach turns out to be both effective and parameter efficient in aggregating temporal information. In the 2nd Youtube-8M video understanding challenge, a single NeXtVLAD model with less than 80M parameters achieves a GAP score of 0.87846 in private leaderboard. A mixture of 3 NeXtVLAD models results in 0.88722, which is ranked 3rd over 394 teams. The code is publicly available at https://github.com/linrongc/youtube-8m.

Proceedings of the 27th ACM International Conference on Multimedia

“…Without rules for logical reasoning, many approaches often employ hand-crafted [19,24,34,43] or deeplearned features [8,9,23,36,44,45] of appearance and motion for action recognition. Recently, researchers attempt to use the semantic-level state changes [1,7,10,25,49,50] for video analysis. For example, Liu et al [25] adopted unary fluents to represent attributes of a single object, and binary fluents for two objects in egocentric videos, and then they used LSTM [11] to recognize which action is performed.…”

Section: Related Workmentioning

confidence: 99%

Explainable Video Action Reasoning via Prior Knowledge and State Transitions

Zhuo

Cheng

Zhang

et al. 2019

Human action analysis and understanding in videos is an important and challenging task. Although substantial progress has been made in past years, the explainability of existing methods is still limited. In this work, we propose a novel action reasoning framework that uses prior knowledge to explain semantic-level observations of video state changes. Our method takes advantage of both classical reasoning and modern deep learning approaches. Specifically, prior knowledge is defined as the information of a target video domain, including a set of objects, attributes and relationships in the target video domain, as well as relevant actions defined by the temporal attribute and relationship changes (i.e. state transitions). Given a video sequence, we first generate a scene graph on each frame to represent concerned objects, attributes and relationships. Then those scene graphs are linked by tracking objects across frames to form a spatio-temporal graph (also called video graph), which represents semantic-level video states. Finally, by sequentially examining each state transition in the video graph, our method can detect and explain how those actions are executed with prior knowledge, just like the logical manner of thinking by humans. Compared to previous works, the action reasoning results of our method can be explained by both logical rules and semantic-level observations of video content changes. Besides, the proposed method can be used to detect multiple concurrent actions with detailed information, such as who (particular objects), when (time), where (object locations) and how (what kind of changes). Experiments on a re-annotated dataset CAD-120 show the effectiveness of our method.