Deep Learning on Lie Groups for Skeleton-Based Action Recognition

Abstract

In recent years, skeleton-based action recognition has become a popular 3D classification problem. State-of-theart methods typically first represent each motion sequence as a high-dimensional trajectory on a Lie group with an additional dynamic time warping, and then shallowly learn favorable Lie group features. In this paper we incorporate the Lie group structure into a deep network architecture to learn more appropriate Lie group features for 3D action recognition. Within the network structure, we design rotation mapping layers to transform the input Lie group features into desirable ones, which are aligned better in the temporal domain. To reduce the high feature dimensionality, the architecture is equipped with rotation pooling layers for the elements on the Lie group. Furthermore, we propose a logarithm mapping layer to map the resulting manifold data into a tangent space that facilitates the application of regular output layers for the final classification. Evaluations of the proposed network for standard 3D human action recognition datasets clearly demonstrate its superiority over existing shallow Lie group feature learning methods as well as most conventional deep learning methods.

NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding

“…Beside RNNs and CNNs, some other deep models have also been introduced for 3D human action recognition. Huang et al [62] incorporated Lie group structure into a deep architecture for skeleton-based action recognition. Tang et al [77] applied deep progressive reinforcement learning to distil the informative frames in the video sequences.…”

Skeleton-Based Hand Gesture Recognition by Learning SPD Matrices with Neural Networks

“…The batch size and the learning rate are set to 30 and 0.01, respectively. The rectification threshold used for the ReEig layer is set to 0.0001 [12]. For gesture classification, we use the LIBLINEAR library [9] with L2regularized L2-loss (dual) to train the classifier and use the default parameter settings in LIBLINEAR (C is set to 1, the tolerance of termination criterion is set to 0.1 and no bias term is added).…”

“…Manifold learning approaches operated on different manifolds, e.g. SPD manifolds [11], Lie groups [12], and Grassmann manifolds [13]. Graph learning approaches [25], [34] were based on graph convolution filters and Deep Residual Networks [9] which have shown very promising results on large scale datasets.…”

A Neural Network Based on SPD Manifold Learning for Skeleton-Based Hand Gesture Recognition

“…Recently, deep learning on manifolds and graphs has increasingly attracted attention. Approaches following this line of research have also been successfully applied to skeleton-based action recognition [19,20,23,27,56]. By extending classical operations like convolutions to manifolds and graphs while respecting the underlying geometric structure of data, they have demonstrated superior performance over other approaches.…”