Multi‐stream adaptive spatial‐temporal attention graph convolutional network for skeleton‐based action recognition

Abstract

Skeleton-based action recognition algorithms have been widely applied to human action recognition. Graph convolutional networks (GCNs) generalize convolutional neural networks (CNNs) to non-Euclidean graphs and achieve significant performance in skeleton-based action recognition. However, existing GCN-based models have several issues, such as the topology of the graph is defined based on the natural skeleton of the human body, which is fixed during training, and it may not be applied to different layers of the GCN model and diverse datasets. Besides, the higher-order information of the joint data, for example, skeleton and dynamic information is not fully utilised. This work proposes a novel multi-stream adaptive spatial-temporal attention GCN model that overcomes the aforementioned issues. The method designs a learnable topology graph to adaptively adjust the connection relationship and strength, which is updated with training along with other network parameters. Simultaneously, the adaptive connection parameters are utilised to optimise the connection of the natural skeleton graph and the adaptive topology graph. The spatial-temporal attention module is embedded in each graph convolution layer to ensure that the network focuses on the more critical joints and frames. A multi-stream framework is built to integrate multiple inputs, which further improves the performance of the network. The final network achieves state-of-the-art performance on both the NTU-RGBD and Kinetics-Skeleton action recognition datasets. The simulation results prove that the proposed method reveals better results than existing methods in all perspectives and that shows the superiority of the proposed method.

Involution Feature Extraction Network Based Human Posture Recognition in Martial Arts Movement Recognition

“…Multiplying this kernel with the feature vector adjacent to the point gives the K K C  feature map, and finally the K K C  kernel is superimposed to obtain the final output feature map, with Involution generating different kernels for different locations and sharing a single kernel at the same location on the channel [12]. The traditional Convolution kernel counts and Involution counts are shown in Equation (1).…”

“…In Equation (12), N is the number of nodes in the network and U is the set of all nodes in the network.…”

Shifting Perspective to See Difference: A Novel Multi-view Method for Skeleton based Action Recognition

“…EfficientGCN [35] considered joints, bone and velocity to increase the capacity of their model. Furthermore, joints motion and bones motion are added by [44] as extra streams in the feature learning pipeline. However, the view stream has not been formally proposed and is well designed in the field due to our best knowledge.…”

“…For instance, some of their work consider the different scale of receptive field [6], capturing both the short-term trajectory and long-term trajectory [5], or design multi-scale ST-GCN [42]. Another group of trending approaches [32,35,44] leverage the capacity of GCN models by introducing multiple streams. They usually take joints (relative and absolute), velocity (joints motion and bones motion) and bones as inputs and fuse all the features together as the final representation.…”