Part-wise Spatio-temporal Attention Driven CNN-based 3D Human Action Recognition

Abstract

Recently, human activity recognition using skeleton data is increasing due to its ease of acquisition and finer shape details. Still, it suffers from a wide range of intra-class variation, inter-class similarity among the actions and view variation due to which extraction of discriminative spatial and temporal features is still a challenging problem. In this regard, we present a novel Residual Inception Attention Driven CNN (RIAC-Net) Network, which visualizes the dynamics of the action in a part-wise manner. The complete skeletonis partitioned into five key parts: Head to Spine, Left Leg, Right Leg, Left Hand, Right Hand. For each part, a Compact Action Skeleton Sequence (CASS) is defined. Part-wise skeleton-based motion dynamics highlights discriminative local features of the skeleton that helps to overcome the challenges of inter-class similarity and intra-class variation with improved recognition performance. The RIAC-Net architecture is inspired by the concept of inception-residual representation that unifies the Attention Driven Residues (ADR) with inception-based Spatio-Temporal Convolution Features (STCF) to learn efficient salient action features. An ablation study is also carried out to analyze the effect of ADR over simple residue-based action representation. The robustness of the proposed framework is evaluated by performing an extensive experiment on four challenging datasets: UT Kinect Action 3D, Florence 3D action, MSR Daily Action3D, and NTU RGB-D datasets, which consistently demonstrate the superiority of the proposed method over other state-of-the-art methods.

Motion-Driven Spatial and Temporal Adaptive High-Resolution Graph Convolutional Networks for Skeleton-Based Action Recognition

“…To the best of our knowledge, so far, no GCN-based method has been evaluated on it. We compare our methods with two publicly available classical GCN-based methods (i.e., ST-GCN [25] and 2s-AGCN [26]), and the SOTA CNN-based methods (e.g., RIAC-LSTM [23], SPMFs [78]), point cloud-based methods (e.g., SequentialPointNet [56], P4Transformer [55]), and…”

“…Year Accuracy(%) ST-GCN [25] 2018 83.27 SPMFs [78] 2018 98.05 2s-AGCN [26] 2019 88.36 MeteorNet [79] 2019 88.5 UnifiedDeep [80] 2019 97.98 Movement polygon [13] 2020 94.13 P4Transformer [55] 2021 90.94 PSTNet [81] 2021 91.2 MMDNN [82] 2021 91.94 RIAC-LSTM [23] 2021 98.06 Complex Network+LSTM [83] 2022 90.7 SequentialPointNet [56] 2022 91.94 2s-MS&TA-HGCN-FC(ours) 90.54 4s-MS&TA-HGCN-FC(ours) 92. 73 depth-based methods (e.g., MMDNN [82]).…”

“…A multi-stream CNN network is used to combine them for classification. Dhiman et al [23] projected a skeleton sequence onto an image and encrypted the human poses using different colors.The pseudo image is then classified using a part-wise spatio-temporal attentiondriven CNN network. Xu et al [24] proposed a pure CNN architecture named topology-aware CNN (Ta-CNN) with a cross-channel feature augmentation module.…”

“…Each action was performed three times by every subject. In the experiment, we follow the recent literature [23], [55], [56] and use the cross-subject evaluation protocol where half of the subjects are used for training and the rest are used for testing. We use the pretrained GCN models on NTU-RGB+D datasets to funtune for evaluation because of the insufficient training samples of this dataset.…”

Fusing angular features for skeleton‐based action recognition using multi‐stream graph convolution network

“…CNN models can improve the accuracy of the model by converting the skeleton data into a pseudo-image form suitable for CNN processing and by adaptively optimizing the loss function and extracting the important features [10][11][12]. As a result, researchers have started to utilize recurrent neural networks for recognition tasks [13,14].…”