Research on Discriminative Skeleton-Based Action Recognition in Spatiotemporal Fusion and Human-Robot Interaction

Abstract

A novel posture motion-based spatiotemporal fused graph convolutional network (PM-STGCN) is presented for skeleton-based action recognition. Existing methods on skeleton-based action recognition focus on independently calculating the joint information in single frame and motion information of joints between adjacent frames from the human body skeleton structure and then combine the classification results. However, that does not take into consideration of the complicated temporal and spatial relationship of the human body action sequence, so they are not very efficient in distinguishing similar actions. In this work, we enhance the ability of distinguishing similar actions by focusing on spatiotemporal fusion and adaptive feature extraction for high discrimination information. Firstly, the local posture motion-based attention (LPM-TAM) module is proposed for the purpose of suppressing the skeleton sequence data with a low amount of motion in the temporal domain, and the representation of motion posture features is concentrated. Besides, the local posture motion-based channel attention module (LPM-CAM) is introduced to make use of the strongly discriminative representation between different action classes of similarity. Finally, the posture motion-based spatiotemporal fusion (PM-STF) module is constructed which fuses the spatiotemporal skeleton data by filtering out the low-information sequence and enhances the posture motion features adaptively with high discrimination. Extensive experiments have been conducted, and the results demonstrate that the proposed model is superior to the commonly used action recognition methods. The designed human-robot interaction system based on action recognition has competitive performance compared with the speech interaction system.

Skeleton Graph-Neural-Network-Based Human Action Recognition: A Survey

“…ST-GCN is a typical spatiotemporal approach since it performs GCN on spatiotemporal graph (STG) directly and therefore extracts spatiotemporal information simultaneously. Methods, such as [ 29 , 48 , 54 , 60 , 68 , 82 , 86 , 96 , 101 , 102 , 103 , 104 , 105 , 106 , 107 , 108 , 109 , 110 , 111 , 112 , 113 ] are all developed based on ST-GCN. Methods based on AGCN also work on STG, such as [ 66 , 73 , 93 , 114 ].…”

“…Q.B. Zhong et al [ 109 ] emphasized the joints with more motions and propose a novel local posture motion-based attention module (LPM-TAM) to filter out low motion information in temporal domain. This operation helps improve the ability of motion-related feature extraction.…”

“…Most papers [ 73 , 74 , 97 , 109 , 126 ] calculated the temporal attention map with the most popular recipe, which adopts a sequence consisted of a pooling layer, a fully connection (FC) layer or one 1DCNN layer, followed by Relu or softmax activation functions. For example, L. Shi et al [ 73 ] performed attention by 1DCNN.…”

“…Q.B. Zhong et al [ 109 ] processed the motion feature map by local posture motion-based temporal attention module and further by local posture motion-based channel attention module, with an aim at selecting the strongest discriminative representations between different posture movements.…”

Human Motion Posture Detection Algorithm Using Deep Reinforcement Learning

“…where p means the similarity coefficient, which is between 0 and 1. e larger the value of p, the higher the similarity between the human motion posture feature and candidate area posture feature [26,27]. When p reaches the maximum value, the candidate area of the human motion posture will become the human motion posture feature to be solved in this frame of image ρ.…”