Exploiting deep residual networks for human action recognition from skeletal data

Abstract

The computer vision community is currently focusing on solving action recognition problems in real videos, which contain thousands of samples with many challenges. In this process, Deep Convolutional Neural Networks (D-CNNs) have played a significant role in advancing the state-of-the-art in various vision-based action recognition systems. Recently, the introduction of residual connections in conjunction with a more traditional CNN model in a single architecture called Residual Network (ResNet) has shown impressive performance and great potential for image recognition tasks. In this paper, we investigate and apply deep ResNets for human action recognition using skeletal data provided by depth sensors. Firstly, the 3D coordinates of the human body joints carried in skeleton sequences are transformed into image-based representations and stored as RGB images. These color images are able to capture the spatial-temporal evolutions of 3D motions from skeleton sequences and can be efficiently learned by D-CNNs. We then propose a novel deep learning architecture based on ResNets to learn features from obtained color-based representations and classify them into action classes. The proposed method is evaluated on three challenging benchmark datasets including MSR Action 3D, KARD, and NTU-RGB+D datasets. Experimental results demonstrate that our method achieves state-of-the-art performance for all these benchmarks whilst requiring less computation resource. In particular, the proposed method surpasses previous approaches by a significant margin of 3.4% on MSR Action 3D dataset, 0.67% on KARD dataset, and 2.5% on NTU-RGB+D dataset. (Huy-Hieu Pham) challenging task due to many obstacles such as viewpoint, occlusion or lighting conditions (Poppe, 2010).Traditional studies on HAR mainly focus on the use of handcrafted local features such as Cuboids (Dollár et al., 2005) or HOG/HOF (Laptev et al., 2008) that are provided by 2D cameras. These approaches typically recognize human actions based on the appearance and movements of human body parts in videos. Another approach is to use Genetic Programming (GP) for generating spatio-temporal descriptors of motions . However, one of the major limitations of the 2D data is the absence of 3D structure from the scene. There-arXiv:1803.07781v1 [cs.CV]

A Deep Learning Approach for Real-Time 3D Human Action Recognition from Skeletal Data

“…Generally speaking, four hypotheses that motivate us to build a skeleton-based representation and design DenseNets for 3D HAR include: (1) human actions can be correctly represented via movements of the skeleton [16]; (2) spatio-temporal evolutions of skeletons can be transformed into color images -a kind of 3D tensor that can be effectively learned by D-CNNs [1,5,3]. This hypothesis was proved in our previous studies [27,28,29]; (3) compared to RGB and depth modalities, skeletal data has high-level information with much less complexity. This makes the learning model much simpler and requiring less computation, allowing us to build real-time deep learning framework for HAR task; (4) DenseNet is currently one of the most effective CNN architecture for image recognition.…”

“…One of the major challenges in exploiting D-CNNs for skeleton-based action recognition is how a skeleton sequence could be effectively represented and fed to the deep networks. As D-CNNs work well on still images [18], our idea therefore is to encode the spatial and temporal dynamics of skeletons into 2D images [28,29]. Two essential elements for describing an action are static poses and their temporal dynamics.…”

A Unified Deep Framework for Joint 3D Pose Estimation and Action Recognition from a Single RGB Camera

“…These intermediate representations are then fed to a Deep Convolutional Neural Network (D-CNNs) for learning and classifying actions. This idea has been proven effective in [51,52,53]. Thus, the spatio-temporal patterns of a 3D pose sequence are transformed into a single color image as a global representation called Enhanced-SPMF [53] via two important elements of a human movement: 3D poses and their motions.…”

A Machine Learning Approach to Detect Violent Behaviour from Video

“…For video object detection and classification, several other neural networks have been proposed. The ResNet architecture uses RGB images with encoded spatial-temporal features extracted from 3D skeleton keypoints [12,21]. In 2018, the ResNet model was extended, under five distinct architectures, in [22].…”