SkeleMotion: A New Representation of Skeleton Joint Sequences based on Motion Information for 3D Action Recognition

Abstract

Due to the availability of large-scale skeleton datasets, 3D human action recognition has recently called the attention of computer vision community. Many works have focused on encoding skeleton data as skeleton image representations based on spatial structure of the skeleton joints, in which the temporal dynamics of the sequence is encoded as variations in columns and the spatial structure of each frame is represented as rows of a matrix. To further improve such representations, we introduce a novel skeleton image representation to be used as input of Convolutional Neural Networks (CNNs), named SkeleMotion. The proposed approach encodes the temporal dynamics by explicitly computing the magnitude and orientation values of the skeleton joints. Different temporal scales are employed to compute motion values to aggregate more temporal dynamics to the representation making it able to capture longrange joint interactions involved in actions as well as filtering noisy motion values. Experimental results demonstrate the effectiveness of the proposed representation on 3D action recognition outperforming the state-of-the-art on NTU RGB+D 120 dataset.

Deep learning‐based action recognition with 3D skeleton: A survey

“…Although RNN-based approaches present excellent results, such methods lack the ability to efficiently learn the spatial relations between the skeleton joints. [9,23,[39][40][41]43,44] In order to explicitly explore spatial information, many researchers encode the skeleton joints to multiple 2D pseudo-images. The representations used to feed CNN-based approaches present the advantage of having the natural ability to learn structural information from 2D arrays (i.e., they learn spatial relations from the skeleton joints).…”

“…The process of SkeleMotion representation. [43] XING AND ZHU of the graph into three parts, the first part represents the physical structure of the human body, the elements in the second part are parameterized and optimized together with other parameters in the training data, and the third part can learn a unique graph for each sample of the dataset. Due to the work [52] proved bone information (the directions and lengths of bones) has a good modality for skeleton-based action recognition.…”

Whole and Part Adaptive Fusion Graph Convolutional Networks for Skeleton-Based Action Recognition

“…The CNN-based methods transform 3D-skeleton sequence data from a vector sequence to a pseudo-image and use the CNN network to extract the pseudo-image features [ 18 , 19 , 20 , 21 , 22 , 23 ]. The advantage of the CNN method lies in its powerful feature extraction capabilities, but it cannot handle the spatial and temporal relationships in skeleton information well.…”

“…With the continuous development of deep learning, many methods extract motion patterns to form a skeleton sequence and use RNNs to model them [ 13 , 14 , 15 , 16 , 17 ]. Other methods based on CNNs transform skeleton data into pseudo images and send them into CNNs for prediction [ 18 , 19 , 20 , 21 , 22 , 23 ]. However, these methods cannot capture the inherent spatial relationship between joints.…”

Generative Model for Skeletal Human Movements based on conditional DC-GAN applied to pseudo-images

“…Liu et al [15] used an enhanced skeleton visualization to represent the skeleton data. SkeleMotion [16] directly encodes data by using orientation and magnitude to provide information regarding the velocity of the movement in different temporal scales. The Tree Structure Reference Joint Image (TSRJI) of Caetano et al [17] combines the use of reference joints and a tree structure skeleton: while the former incorporates different spatial relationships between the joints, the latter preserves important spatial relations by traversing a skeleton tree with a depth-first order algorithm.…”