Explainable Video Action Reasoning via Prior Knowledge and State Transitions

Wang

IEEE Trans. on Image Process.

et al. 2022

117

Attempt to fully explore the fine-grained temporal structure and global-local chronological characteristics for selfsupervised video representation learning, this work takes a closer look at exploiting the temporal structure of videos and further proposes a novel self-supervised method named Temporal Contrastive Graph (TCG). In contrast to the existing methods that randomly shuffle the video frames or video snippets within a video, our proposed TCG roots in a hybrid graph contrastive learning strategy to regard the intersnippet and intra-snippet temporal relationships as selfsupervision signals for temporal representation learning. To increase the temporal diversity of features more comprehensively and precisely, our proposed TCG integrates the prior knowledge about the frame and snippet orders into temporal contrastive graph structures, i.e., the intra-/inter-snippet temporal contrastive graph modules. By randomly removing edges and masking node features of the intra-snippet graphs or inter-snippet graphs, our TCG can generate different correlated graph views. Then, specific contrastive losses are designed to maximize the agreement between node embeddings in different views. To learn the global context representation and recalibrate the channelwise features adaptively, we introduce an adaptive video snippet order prediction module, which leverages the relational knowledge among video snippets to predict the actual snippet orders.Experimental results demonstrate the superiority of our TCG over the state-of-the-art methods on largescale action recognition and video retrieval benchmarks.

Section: Introductionmentioning

confidence: 99%

TCGL: Temporal Contrastive Graph for Self-Supervised Video Representation Learning

Wang

IEEE Trans. on Image Process.

et al. 2022

117

“…Understanding these movements is vital for an artificial intelligent agent to comprehend and interact with the ever-changing world. Studies on social behavior analysis [8,9], action recognition [63,58], and video summarizing [59] have also acknowledged the importance of movement.…”

Section: Introductionmentioning

confidence: 99%

Unsupervised Motion Representation Learning with Capsule Autoencoders

Xu,

Shen,

Wong

et al. 2021

Preprint

Self Cite

We propose the Motion Capsule Autoencoder (MCAE), which addresses a key challenge in the unsupervised learning of motion representations: transformation invariance. MCAE models motion in a two-level hierarchy. In the lower level, a spatio-temporal motion signal is divided into short, local, and semanticagnostic snippets. In the higher level, the snippets are aggregated to form fulllength semantic-aware segments. For both levels, we represent motion with a set of learned transformation invariant templates and the corresponding geometric transformations by using capsule autoencoders of a novel design. This leads to a robust and efficient encoding of viewpoint changes. MCAE is evaluated on a novel Trajectory20 motion dataset and various real-world skeleton-based human action datasets. Notably, it achieves better results than baselines on Trajectory20 with considerably fewer parameters and state-of-the-art performance on the unsupervised skeleton-based action recognition task.

“…Besides, the logical rules of different RPM problems are often different, which makes this task more challenging. With the success of deep learning in image [7], [8], [9] and videos [10], [11], [12], [13], [14], recent approaches [3], [4], [15], [16], [17], [18] try to solve RPM problems with deep neural networks. Similar to the image and video classification, these methods also address abstract reasoning on RPM problems as a classification task.…”

Section: Introductionmentioning

confidence: 99%

Unsupervised Abstract Reasoning for Raven's Problem Matrices

Zhuo¹,

Huang²,

Kankanhalli³

2021

Preprint

Self Cite

Raven's Progressive Matrices (RPM) is highly correlated with human intelligence, and it has been widely used to measure the abstract reasoning ability of humans. In this paper, to study the abstract reasoning capability of deep neural networks, we propose the first unsupervised learning method for solving RPM problems. Since the ground truth labels are not allowed, we design a pseudo target based on the prior constraints of the RPM formulation to approximate the groundtruth label, which effectively converts the unsupervised learning strategy into a supervised one. However, the correct answer is wrongly labelled by the pseudo target, and thus the noisy contrast will lead to inaccurate model training. To alleviate this issue, we propose to improve the model performance with negative answers. Moreover, we develop a decentralization method to adapt the feature representation to different RPM problems. Extensive experiments on three datasets demonstrate that our method even outperforms some of the supervised approaches. Our code is available at https://github.com/visiontao/ncd.