Fusing Higher-Order Features in Graph Neural Networks for Skeleton-Based Action Recognition

Qin, Zhenyue; Liu, Yang; Pan, Jing; Kim, Dongwoo; Wang, Lei; McKay, Bob; Anwar, Saeed; Gedeon, Tom

doi:10.1109/tnnls.2022.3201518

Cited by 53 publications

(18 citation statements)

References 39 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Caetano et al (2019a) performed feature extraction by calculating the size and direction values of the skeleton to recode the skeleton sequence and map it into a picture. Qin et al (2021) combined spatio-temporal graph convolutional networks to extract higher-order features of the action by defining several special angular encodings between joints. Referring to the above studies, we first associate the possibility based on edge-level, and secondly, the human edge-vector exists in different directions, and the design of higher-order features in different directions considering the motion features of different directions.…”

Section: Feature Definition For Skeleton-based Human Action Recognitionmentioning

confidence: 99%

Direction-guided two-stream convolutional neural networks for skeleton-based action recognition

Zhang

Sun

et al. 2023

Soft Comput

View full text Add to dashboard Cite

In skeleton-based action recognition, the approach of treating skeleton data as pseudo-images using convolutional neural networks (CNN) has proven to be effective. However, among the existing CNN-based approaches, most of them focus on modeling information at the joint-level ignoring the size and direction information of the skeleton edges, which play an important role in action recognition, and these approaches may not be optimal. In addition, combining the directionality of human motion to portray the motion variations information of the action, which is more natural and reasonable for action sequence modeling, is rarely considered in existing approaches. In this work, we propose a novel direction-guided two-stream convolutional neural networks (DG-2sCNN) for skeleton-based action recognition. On the first stream, our model focuses on our defined edge-level information (including edge and edge motion information) with directionality in the skeleton data to explore the spatio-temporal features of the action. On the second stream, since the motion is directional, we define different skeleton edge directions and extract different motion information (including translation and rotation information) in different directions in order to better exploit the motion features of the action. Besides, we propose the description of human motion inscribed by a combination of translation and rotation, and explore the way they are integrated. We conducted extensive experiments on two challenging datasets, NTU-RGB+D 60 and NTU-RGB+D 120, to verify the superiority of our proposed method over state-of-the-art methods. The experimental results demonstrate that the proposed direction-guided edge-level information and motion information complement each other for better action recognition.

show abstract

Section: Feature Definition For Skeleton-based Human Action Recognitionmentioning

confidence: 99%

Direction-guided two-stream convolutional neural networks for skeleton-based action recognition

Zhang

Sun

et al. 2023

Soft Comput

View full text Add to dashboard Cite

show abstract

“…However, due to the need to calculate multiple branches at the same time, the calculation cost was high. Literature [2,14] is a model mainly based on 3D convolution. It can be seen that the model mainly focuses on the improvement of accuracy, and the model is relatively complex and the calculation is slow.…”

Section: Comparison With Other Action Recognition Algorithmsmentioning

confidence: 99%

“…Literature [7] and literature [10] have computational costs similar to that of this paper, but the accuracy is 5.3% and 0.6% lower than that of this paper, respectively. Literature [11][12][13][14][15][16] has a higher performance in recognition accuracy, but also has a larger amount of computation and parameters. Therefore, the method proposed in this paper balances the three performance indexes of calculation cost, parameter number and recognition accuracy well, and ensures a high recognition accuracy while reducing parameter number and calculation cost.…”

Section: Comparison With Other Action Recognition Algorithmsmentioning

confidence: 99%

Lightweight skeleton-based action recognition research

Wang

2023

3rd International Conference on Artificial Intelligence, Automation, and High-Performance Computing (AIAHPC 2023)

View full text Add to dashboard Cite

Action recognition is a cutting-edge research direction in the field of computer vision, widely used in video surveillance, video retrieval, human-computer interaction, and other fields. However, existing action recognition algorithms improve prediction accuracy at the cost of having too many model parameters and high computational complexity, which limits their practical application. In order to address this issue and improve recognition accuracy with higher computational efficiency, we propose a dense residual 3D depth-separable convolutional network model based on channel attention mechanism. By extending the depth separable convolution to 3D, the model can maintain high accuracy while greatly reducing the number of model parameters and operational complexity. The dense residual structure is introduced to enhance the training ability of the network, improve feature extraction, and further improve network performance by learning the correlation between channels through the channel attention mechanism. Compared with other action recognition algorithms on two datasets, NTU RGB-D and Fall Dataset Lei2, the proposed action recognition algorithm has better robustness under lightweight condition, with 415.9 MFLOPs of computation, 205.4k of parameters, and 90.5% of CS, 95.5% of CV and 95.97% of accuracy, respectively. The proposed algorithm has a high recognition accuracy and greatly reduces the calculation cost, which can provide better performance for practical applications.

show abstract

“…In recent years, researchers have become increasingly interested in GCN-based methods [35]- [39] which can reflect the structured relationships between skeletons. Most studies have focused on spatial modeling, which contains pre-defined [7], learnable [8], and dynamic [11] ways.…”

Section: A Skeleton-based Action Recognitionmentioning

confidence: 99%

Spatial-Temporal Attention Network with Multi-similarity Loss for Fine-Grained Skeleton-Based Action Recognition

Liu

et al. 2021

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Graph convolutional networks have been widely used in skeleton-based action recognition. However, existing approaches are limited in fine-grained action recognition due to the similarity of inter-class data. Moreover, the noisy data from pose extraction increases the challenge of fine-grained recognition. In this work, we propose a flexible attention block called Channel-Variable Spatial-Temporal Attention (CVSTA) to enhance the discriminative power of spatial-temporal joints and obtain a more compact intra-class feature distribution. Based on CVSTA, we construct a Multi-Dimensional Refinement Graph Convolutional Network (MDR-GCN), which can improve the discrimination among channel-, joint-and frame-level features for fine-grained actions. Furthermore, we propose a Robust Decouple Loss (RDL), which significantly boosts the effect of the CVSTA and reduces the impact of noise. The proposed method combining MDR-GCN with RDL outperforms the known stateof-the-art skeleton-based approaches on fine-grained datasets, FineGym99 and FSD-10, and also on the coarse dataset NTU-RGB+D X-view version.

show abstract

Fusing Higher-Order Features in Graph Neural Networks for Skeleton-Based Action Recognition

Cited by 53 publications

References 39 publications

Direction-guided two-stream convolutional neural networks for skeleton-based action recognition

Direction-guided two-stream convolutional neural networks for skeleton-based action recognition

Lightweight skeleton-based action recognition research

Spatial-Temporal Attention Network with Multi-similarity Loss for Fine-Grained Skeleton-Based Action Recognition

Contact Info

Product

Resources

About