A comparative review of graph convolutional networks for human skeleton-based action recognition

Feng, Liqi; Zhao, Yaqin; Zhao, Wenxuan; Tang, Jiaxi

doi:10.1007/s10462-021-10107-y

Cited by 27 publications

(12 citation statements)

References 67 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The graph convolutional network uses a graph to save and transfer data and an adjacency matrix to save the connections between the nodes. Thereby, graph convolutional network has an advantage over other deep learning-based methods when dealing with graph data tasks such as skeleton-based action recognition [16][17][18]. In the literature [19], Yan et al propose a spatio-temporal graph convolutional network (ST-GCN) named.…”

Section: Related Workmentioning

confidence: 99%

Fusing angular features for skeleton‐based action recognition using multi‐stream graph convolution network

Huang,

Liu,

Shang

et al. 2024

IET Image Processing

View full text Add to dashboard Cite

Distinguishing similar actions has been a challenging challenge in skeleton‐based action recognition. Since the joint coordinates in these actions are similar, it is difficult to accomplish the recognition task using traditional joint features. To address this issue, the use of angle features to capture subtle nuances in various body parts, along with a critical angle enhancement module that assigns weights to different angle feature representations for a given action are proposed, highlighting the critical angle feature representation. The approach is evaluated using a three‐stream ensemble method on three large action recognition datasets, NTU‐RGB+D, NTU‐RGB+D 120, and Kinetics‐400. The experimental results demonstrate that incorporating angular information can effectively complement joint and skeletal features, leading to improved recognition of similar actions and enhanced model performance and robustness.

show abstract

Section: Related Workmentioning

confidence: 99%

Fusing angular features for skeleton‐based action recognition using multi‐stream graph convolution network

Huang,

Liu,

Shang

et al. 2024

IET Image Processing

View full text Add to dashboard Cite

show abstract

“…Recently, GCNs have emerged as SOTA approaches that effectively exploit the geometric structures of skeleton graphs to recognize human action and interaction [38]. Yan et al [19] introduce ST-GCN as a pioneering backbone for future GCN-based models.…”

Section: A Gcn-based Human Interaction Recognitionmentioning

confidence: 99%

Distinct Motion-Preserving Graph Convolutional Network for Two-Person Interaction Recognition

Phuong,

Tran,

Lee

2023

IEEE Access

View full text Add to dashboard Cite

Graph Convolutional Networks (GCNs) have gained widespread adoption in modeling human skeleton sequences for two-person interaction recognition. Most GCN-based models achieve state-of-the-art results by leveraging either intra-body or inter-body connections. However, using only intra-body relations may ignore important interactive features between two individuals, whereas relying on inter-body relations may weaken the specific motion dynamics of each skeleton. To address these shortcomings, we propose a Distinct Motion-Preserving GCN (DMP-GCN) that utilizes intra-body and inter-body graphs to extract interactive features from two human bodies while preserving the distinct motion characteristics of each skeleton. Specifically, two motion-specific streams are adopted to capture specific motion features of each human skeleton and an interactive stream is applied to model the interactive dynamics of two bodies. In addition, we introduce a new graph labeling strategy called Distance Variation Labeling which is a datadriven approach for defining the edge strength in the skeleton graph. Extensive experiments show that our proposed approaches outperform state-of-the-art methods on two large-scale human interaction datasets, NTU RGB+D (mutual) and NTU RGB+D 120 (mutual).

show abstract

“…On the other hand, skeleton-based methods predict the action class from the body’s skeletal structure and its motion [15, 11]. They thus rely on an additional preprocessing step called pose estimation , in which the body parts, such as joints and bones, are detected and their coordinates extracted from each video frame [6].…”

Section: Introductionmentioning

confidence: 99%

“…Comparatively, skeleton-based methods are less sensitive to these changes [18] and, given a robust pose estimator, are therefore likely to maintain a high action recognition accuracy. Furthermore, extracting the animal’s pose coordinates from high-dimensional video segments drastically reduces the network’s input space dimensionality, its computational complexity, and overall power consumption [15], which can represent game-changing advantages for field researchers with limited computational resources. Finally, identifying the pose offers a pre-computed geometrical quantification of the animal’s body motion and behavioral changes [67, 45].…”

Section: Introductionmentioning

confidence: 99%

ASBAR: an Animal Skeleton-Based Action Recognition framework. Recognizing great ape behaviors in the wild using pose estimation with domain adaptation

Fuchs,

Genty,

Zuberbühler

et al. 2023

Preprint

View full text Add to dashboard Cite

To date, the investigation and classification of animal behaviors have mostly relied on direct human observations or video recordings with posthoc analysis, which can be labor-intensive, time-consuming, and prone to human bias. Recent advances in machine learning for computer vision tasks, such as pose estimation and action recognition, thus have the potential to significantly improve and deepen our understanding of animal behavior. However, despite the increased availability of open-source toolboxes and large-scale datasets for animal pose estimation, their practical relevance for behavior recognition remains under-explored. In this paper, we propose an innovative framework, ASBAR, for Animal Skeleton-Based Action Recognition, which fully integrates animal pose estimation and behavior recognition. We demonstrate the use of this framework in a particularly challenging task: the classification of great ape natural behaviors in the wild. First, we built a robust pose estimator model leveraging OpenMonkeyChallenge, one of the largest available open-source primate pose datasets, through a benchmark analysis on several CNN models from DeepLabCut, integrated into our framework. Second, we extracted the great ape’s skeletal motion from the PanAf dataset, a large collection of in-the-wild videos of gorillas and chimpanzees annotated for natural behaviors, which we used to train and evaluate PoseConv3D from MMaction2, a second deep learning model fully integrated into our framework. We hereby classify behaviors into nine distinct categories and achieve a Top 1 accuracy of 74.98%, comparable to previous studies using video-based methods, while reducing the model’s input size by a factor of around 20. Additionally, we provide an open-source terminal-based GUI that integrates our full pipeline and release a set of 5,440 keypoint annotations to facilitate the replication of our results on other species and/or behaviors. All models, code, and data can be accessed at: https://github.com/MitchFuchs/asbar.

show abstract

A comparative review of graph convolutional networks for human skeleton-based action recognition

Cited by 27 publications

References 67 publications

Fusing angular features for skeleton‐based action recognition using multi‐stream graph convolution network

Fusing angular features for skeleton‐based action recognition using multi‐stream graph convolution network

Distinct Motion-Preserving Graph Convolutional Network for Two-Person Interaction Recognition

ASBAR: an Animal Skeleton-Based Action Recognition framework. Recognizing great ape behaviors in the wild using pose estimation with domain adaptation

Contact Info

Product

Resources

About