Learning shape and motion representations for view invariant skeleton-based action recognition

Li, Yanshan; Xia, Rongjie; Liu, Xing

doi:10.1016/j.patcog.2020.107293

Cited by 41 publications

(16 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…At present, the deep learning methods for skeleton-based action recognition are mainly divided into three categories: (1) The RNN-based methods [36,30,29,31] is based on the natural time properties of the skeleton sequence, and then modeled by Long Short Term Memory (LSTM), Gated Recurrent Unit (GRU), etc. ; (2) The CNNbased methods [19,12,42,17] usually convert the skeleton sequence into the pseudoimages using specific transformation rules, and model it with efficient image classification networks. The CNN-based methods usually combined with RNN-based methods to model the temporal information of skeleton sequence.…”

Section: Skeleton-based Action Recognitionmentioning

confidence: 99%

Spatio-Temporal Tuples Transformer for Skeleton-Based Action Recognition

Qiu¹,

Hou²,

Ren³

et al. 2022

Preprint

View full text Add to dashboard Cite

Capturing the dependencies between joints is critical in skeletonbased action recognition task. Transformer shows great potential to model the correlation of important joints. However, the existing Transformer-based methods cannot capture the correlation of different joints between frames, which the correlation is very useful since different body parts (such as the arms and legs in "long jump") between adjacent frames move together. Focus on this problem, A novel spatio-temporal tuples Transformer (STTFormer) method is proposed. The skeleton sequence is divided into several parts, and several consecutive frames contained in each part are encoded. And then a spatio-temporal tuples self-attention module is proposed to capture the relationship of different joints in consecutive frames. In addition, a feature aggregation module is introduced between non-adjacent frames to enhance the ability to distinguish similar actions. Compared with the state-of-the-art methods, our method achieves better performance on two large-scale datasets.

show abstract

Section: Skeleton-based Action Recognitionmentioning

confidence: 99%

Spatio-Temporal Tuples Transformer for Skeleton-Based Action Recognition

Qiu¹,

Hou²,

Ren³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Many researchers have spent their valuable time preparing some challenging datasets based on the skeleton of the human body for further experiments including NTU, MSR Action3D, Berkeley MHAD, HDM05, and UTD multimodal human action dataset (UTD-MHAD) datasets. Several ideas developed based on skeleton data [16][17][18][19][20][21][22][23][24][25][26][27][28][29] are used to separate action classes. In [16], J. Imran et al evaluated a method based on skeleton augmented data of 3D skeleton joints information using 3D transformations and designed a RNN-based BiGRU for the purpose of classification.…”

Section: Skeleton-based Action Recognitionmentioning

confidence: 99%

“…For dynamic skeleton sequences, they also proposed a temporal stack learning network. Li et al [26] represented skeleton sequences as a subset of geometric algebra to extract temporal and spatial features such as shape and motion. They also designed a rotor based view transformation method and spatio temporal view-invariant model to eliminate the effect of viewpoint variation and to combine skeleton joints and bones to capture spatial configuration and temporal dynamics, respectively.…”

Section: Skeleton-based Action Recognitionmentioning

confidence: 99%

Deep Learning-Based Action Recognition Using 3D Skeleton Joints Information

2020

View full text Add to dashboard Cite

Human action recognition has turned into one of the most attractive and demanding fields of research in computer vision and pattern recognition for facilitating easy, smart, and comfortable ways of human-machine interaction. With the witnessing of massive improvements to research in recent years, several methods have been suggested for the discrimination of different types of human actions using color, depth, inertial, and skeleton information. Despite having several action identification methods using different modalities, classifying human actions using skeleton joints information in 3-dimensional space is still a challenging problem. In this paper, we conceive an efficacious method for action recognition using 3D skeleton data. First, large-scale 3D skeleton joints information was analyzed and accomplished some meaningful pre-processing. Then, a simple straight-forward deep convolutional neural network (DCNN) was designed for the classification of the desired actions in order to evaluate the effectiveness and embonpoint of the proposed system. We also conducted prior DCNN models such as ResNet18 and MobileNetV2, which outperform existing systems using human skeleton joints information.

show abstract

“…We could still recognize the same action for both actors through the movement of the actors' feet and elbows according to key point shifts. Previous methods used in the skeleton-based action recognition task [31,32] have proven the effectiveness of key points and their shifts for recognizing human actions, where the key points and their correspondence across frames are provided as inputs. Skeleton data outlines the key points of actors in the video, suppressing the influence of trivial environment information during the feature extraction process.…”

Section: Fully-supervised Learningmentioning

confidence: 99%

“…Key points and their displacements have been used mainly in skeleton-based action recognition [10,31,32,59]. Most of the skeleton-based methods take skeleton data as the input, which is generated by devices [33] or pose estimation algorithms [35] in the form of 2D or 3D coordinates.…”

Section: Skeleton-based Action Recognitionmentioning

confidence: 99%

Effective action recognition with fully supervised and self-supervised methods

Cao¹

View full text Add to dashboard Cite

Action recognition in videos has attracted interest in computer vision and machine learning communities thanks to its applications such as surveillance and smart homes. In addition to spatial information in individual frames, videos contain temporal information across the temporal dimension. Therefore, effective spatiotemporal representation is the key to accurate action recognition in videos.Previous works have proposed various fully-supervised and self-supervised methods for video representation learning. For fully-supervised methods, most of them utilize convolution neural networks (CNNs) to extract spatial representation while temporal representation is usually modelled by pixel-wise correlations. However, it is inefficient to extract correlations between all pixels since some of them may relate to in-salient area (e.g. backgrounds or environments). On the other hand, self-supervised methods are proposed to leverage more accessible unlabled data on the Internet and transfer the extracted representation for different downstream tasks. The core of self-supervised methods is to design a pretext task where supervision signal is automatically generated based on characteristics of unlabeled data.Although self-supervised methods avoid the annotation of labeled data, compared to fully-supervised methods, there is room for performance improvement of selfsupervised methods. In this thesis, we address the above research gap with two novel deep learning methods, to advance fully-supervised and self-supervised methods, respectively. For fully-supervised learning, we propose a novel Key Point Shift Embedding Module (KPSEM) to adaptively extract channel-wise key point shifts across video frames without key point annotation for temporal feature extraction. Key points are adaptively extracted as feature points with maximum feature values at split regions, while key point shifts are the spatial displacements of corresponding key points. The key point shifts are encoded as the overall temporal features via linear embedding layers in a multi-set manner. vi To advance self-supervised learning, we propose a novel self-supervised learning method, called Video Incoherence Detection (VID), that leverages incoherence detection for spatio-temporal feature extraction. It roots from the observation that visual systems of human beings can easily identify video incoherence based on their comprehensive understanding of videos. Specifically, the training sample, denoted as the incoherent clip, is constructed by multiple sub-clips hierarchically sampled from the same raw video with various lengths of incoherence between each other. The network is trained to learn high-level representation by predicting the relative location and length of incoherence given the incoherent clip as input. Additionally, intra-video contrastive learning is introduced to maximize the mutual information between different incoherent clips from the same raw video. Our experiments show that both KPSEM and VID achieve state-of-the-art performance on action recognition wi...

show abstract

Learning shape and motion representations for view invariant skeleton-based action recognition

Cited by 41 publications

References 15 publications

Spatio-Temporal Tuples Transformer for Skeleton-Based Action Recognition

Spatio-Temporal Tuples Transformer for Skeleton-Based Action Recognition

Deep Learning-Based Action Recognition Using 3D Skeleton Joints Information

Effective action recognition with fully supervised and self-supervised methods

Contact Info

Product

Resources

About