Skeleton-Based Action Recognition With Directed Graph Neural Networks

Shi, Lei; Zhang, Yifan; Cheng, Jian; Lu, Hanqing

doi:10.1109/cvpr.2019.00810

Cited by 788 publications

(537 citation statements)

References 22 publications

Supporting

Mentioning

536

Contrasting

Unclassified

Order By: Relevance

“…We achieve better results than the current state of the art on the CS benchmark (91.8%) with 1.9% accuracy increase. On the CV benchmark, results are comparable (91.8±0.40% for FU-SION against 91.8±0.40% for DGNN [41]). We conclude to the efficacy of IR data to correctly interpret human actions.…”

Section: ) Comparison With the State Of The Artmentioning

confidence: 87%

“…In [42], the length and direction of bones are used in addition to joint coordinates while adapting the topology of the graph. Shi et al represent skeleton data as a directed acyclic graph based on kinematic dependencies of joints and bones [41]. GCNs report the current state-of-the-art results on benchmark datasets.…”

Section: Related Work a Skeleton-based Approachesmentioning

confidence: 99%

See 1 more Smart Citation

Infrared and 3D Skeleton Feature Fusion for RGB-D Action Recognition

Boissiere

Noumeir

2020

IEEE Access

View full text Add to dashboard Cite

For skeleton-based action recognition from depth cameras, distinguishing object-related actions with similar motions is a difficult task. The other available video streams (RGB, infrared, depth) may provide additional clues, given an appropriate feature fusion strategy. We propose a modular network combining skeleton and infrared data. A pre-trained 2D convolutional neural network (CNN) is used as a pose module to extract features from skeleton data. A pre-trained 3D CNN is used as an infrared module to extract visual features from videos. Both feature vectors are then fused and exploited jointly using a multilayer perceptron (MLP). The 2D skeleton coordinates are used to crop a region of interest around the subjects for the infrared videos. Infrared is favored over RGB, as it is less affected by illumination conditions and usable in the dark. We are the first to combine infrared and skeleton data. We evaluate our method on the NTU RGB+D dataset, the largest dataset for human action recognition from depth cameras. We perform extensive ablation studies. In particular, we show the strong contributions of our cropping strategy and pre-training on action classification accuracy. We also test various feature fusion schemes. Feature sum on an element-wise level yields the best results. Our method achieves state-of-the-art performances on NTU RBG+D.

show abstract

Section: ) Comparison With the State Of The Artmentioning

confidence: 87%

Section: Related Work a Skeleton-based Approachesmentioning

confidence: 99%

Infrared and 3D Skeleton Feature Fusion for RGB-D Action Recognition

Boissiere

Noumeir

2020

IEEE Access

View full text Add to dashboard Cite

show abstract

“…Although 2s Shift-GCN [ 43 ] performs slightly better on Cross-subject benchmark of the NTU-RGB+D than our model, it is tailored for the 3D skeleton and can not be effectively applied to the 2D skeleton. The accuracy of DGNN [ 41 ] on Cross-subject benchmark of the NTU-RGB+D is slightly higher than our model by 0.4%, but the DGNN uses four input data streams while our model only uses two input data streams, and its calculation cost is 16.4 times more than our model.…”

Section: Methodsmentioning

confidence: 94%

“…In the Kinetics dataset, we compare our model with eight state-of-the-art approaches. These eight approaches can be divided into four categories: traditional method [ 44 ], LSTM-based method [ 17 ], CNN-based method [ 24 ], and GCN-based methods [ 4 , 6 , 7 , 41 , 42 ]. Table 2 presents the top-1 and top-5 classification performances.…”

Section: Methodsmentioning

confidence: 99%

Global Co-Occurrence Feature and Local Spatial Feature Learning for Skeleton-Based Action Recognition

Xie

Xin

Liu

et al. 2020

Entropy

View full text Add to dashboard Cite

Recent progress on skeleton-based action recognition has been substantial, benefiting mostly from the explosive development of Graph Convolutional Networks (GCN). However, prevailing GCN-based methods may not effectively capture the global co-occurrence features among joints and the local spatial structure features composed of adjacent bones. They also ignore the effect of channels unrelated to action recognition on model performance. Accordingly, to address these issues, we propose a Global Co-occurrence feature and Local Spatial feature learning model (GCLS) consisting of two branches. The first branch, based on the Vertex Attention Mechanism branch (VAM-branch), captures the global co-occurrence feature of actions effectively; the second, based on the Cross-kernel Feature Fusion branch (CFF-branch), extracts local spatial structure features composed of adjacent bones and restrains the channels unrelated to action recognition. Extensive experiments on two large-scale datasets, NTU-RGB+D and Kinetics, demonstrate that GCLS achieves the best performance when compared to the mainstream approaches.

show abstract

“…For example, the 2s-AGCN in [23] contains about 6.94 million parameters, and takes nearly 4 GPU-days for model training on the NTU RGB+D 60 dataset [21]. And the DGNN [22] contains more than 26 million parameters which is very hard for parameter tuning on large-scale datasets. Thus, the high model complexity has seriously limited the development of skeleton-based action recognition, while there are few literatures on this issue.…”

Section: Introductionmentioning

confidence: 99%

Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition

Song

Zhang

Shan

et al. 2020

Proceedings of the 28th ACM International Conference on Multimedia

272

166

View full text Add to dashboard Cite

One essential problem in skeleton-based action recognition is how to extract discriminative features over all skeleton joints. However, the complexity of the State-Of-The-Art (SOTA) models of this task tends to be exceedingly sophisticated and over-parameterized, where the low efficiency in model training and inference has obstructed the development in the field, especially for large-scale action datasets. In this work, we propose an efficient but strong baseline based on Graph Convolutional Network (GCN), where three main improvements are aggregated, i.e., early fused Multiple Input Branches (MIB), Residual GCN (ResGCN) with bottleneck structure and Part-wise Attention (PartAtt) block. Firstly, an MIB is designed to enrich informative skeleton features and remain compact representations at an early fusion stage. Then, inspired by the success of the ResNet architecture in Convolutional Neural Network (CNN), a ResGCN module is introduced in GCN to alleviate computational costs and reduce learning difficulties in model training while maintain the model accuracy. Finally, a PartAtt block is proposed to discover the most essential body parts over a whole action sequence and obtain more explainable representations for different skeleton action sequences. Extensive experiments on two large-scale datasets, i.e., NTU RGB+D 60 and 120, validate that the proposed baseline slightly outperforms other SOTA models and meanwhile requires much fewer parameters during training and inference procedures, e.g., at most 34 times less than DGNN, which is one of the best SOTA methods. CCS CONCEPTS • Computing methodologies → Activity recognition and understanding.

show abstract

Skeleton-Based Action Recognition With Directed Graph Neural Networks

Cited by 788 publications

References 22 publications

Infrared and 3D Skeleton Feature Fusion for RGB-D Action Recognition

Infrared and 3D Skeleton Feature Fusion for RGB-D Action Recognition

Global Co-Occurrence Feature and Local Spatial Feature Learning for Skeleton-Based Action Recognition

Stronger, Faster and More Explainable: A Graph Convolutional Baseline for Skeleton-based Action Recognition

Contact Info

Product

Resources

About