Action recognition based on RGB and skeleton data sets: A survey

Yue, Rujing; Tian, Zhiqiang; Du, Shaoyi

doi:10.1016/j.neucom.2022.09.071

Cited by 39 publications

(13 citation statements)

References 104 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Yang et al. [5] proposed using DMM (Depth Motion Maps) to describe the behaviour's 3D structure and shape information. In order to make use of the different body shapes and motion information in the depth map, each depth frame was projected onto the three orthogonal Descartes planes.…”

Section: Multimodal Cooperative Self‐attention Networkmentioning

confidence: 99%

“…Yang et al [5] proposed using DMM (Depth Motion Maps) to describe the behaviour's 3D structure and shape information.…”

Section: Depth Self-attention Subnetworkmentioning

confidence: 99%

“…Traditional behavior recognition mainly relies on manual extraction of features to identify [4][5], which has shortcomings in generalization performance. In recent years, with the introduction of a series of deep learning frameworks, deep learning has been playing a leading role in the field of behavior recognition.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Multimodal cooperative self‐attention network for action recognition

Zhong

Hou

Liang

et al. 2023

IET Image Processing

View full text Add to dashboard Cite

Multimodal human behaviour recognition is a research hotspot in computer vision. To fully use both skeleton and depth data, this paper constructs a new multimodal network identification scheme combined with the self‐attention mechanism. The system comprises a transformer‐based skeleton self‐attention subnetwork and a depth self‐attention subnetwork based on CNN. In the skeleton self‐attention subnetwork, this paper proposes a motion synergy space feature that can integrate the information of each joint point according to the entirety and synergy of human motion and puts forward a quantitative standard for the contribution degree of each joint motion. In this paper, the results from the skeleton self‐attention subnetwork and the depth self‐attention subnetwork are integrated and they are verified on the NTU RGB+D and UTD‐MHAD datasets. The authors have achieved 90% recognition rate on UTD‐MHAD dataset, and the CS recognition rate of the authors’ method on the NTU RGB+D dataset reaches 90.5% and the recognition rate of CV is 94.7%. Experimental results show that the network structure proposed in this paper achieves a high recognition rate, and its performance is better than most current methods.

show abstract

Section: Multimodal Cooperative Self‐attention Networkmentioning

confidence: 99%

“…Yang et al [5] proposed using DMM (Depth Motion Maps) to describe the behaviour's 3D structure and shape information.…”

Section: Depth Self-attention Subnetworkmentioning

confidence: 99%

See 1 more Smart Citation

Multimodal cooperative self‐attention network for action recognition

Zhong

Hou

Liang

et al. 2023

IET Image Processing

View full text Add to dashboard Cite

show abstract

“…Conventional methods utilize cost-effective modalities, such as RGB-D videos, which provide rich information for recognizing human interaction. Compared to RGB-D data, skeleton data have lower computational costs and are more robust against noises, background changes, and occlusion, making them the optimal choice for many researchers [2], [3]. Numerous pose estimation algorithms have been developed to extract 3D key-point coordinates of human skeletons from videos [4]- [8].…”

Section: Introductionmentioning

confidence: 99%

Distinct Motion-Preserving Graph Convolutional Network for Two-Person Interaction Recognition

Phuong,

Tran,

Lee

2023

IEEE Access

View full text Add to dashboard Cite

Graph Convolutional Networks (GCNs) have gained widespread adoption in modeling human skeleton sequences for two-person interaction recognition. Most GCN-based models achieve state-of-the-art results by leveraging either intra-body or inter-body connections. However, using only intra-body relations may ignore important interactive features between two individuals, whereas relying on inter-body relations may weaken the specific motion dynamics of each skeleton. To address these shortcomings, we propose a Distinct Motion-Preserving GCN (DMP-GCN) that utilizes intra-body and inter-body graphs to extract interactive features from two human bodies while preserving the distinct motion characteristics of each skeleton. Specifically, two motion-specific streams are adopted to capture specific motion features of each human skeleton and an interactive stream is applied to model the interactive dynamics of two bodies. In addition, we introduce a new graph labeling strategy called Distance Variation Labeling which is a datadriven approach for defining the edge strength in the skeleton graph. Extensive experiments show that our proposed approaches outperform state-of-the-art methods on two large-scale human interaction datasets, NTU RGB+D (mutual) and NTU RGB+D 120 (mutual).

show abstract

“…From the view of the intention of the one-pixel-wide lines, skeletons generally are used to represent the shape features and topological structures of the original object [2]. This kind of thinning algorithms usually operates on binary images and is used in the applications of object detection tasks [3,4], action recognition [5][6][7], and remote sensing image analysis [8]. Thinned lines and curve representations obtained from the edges which commonly have a limited number of pixels in the gradient direction are usually used in fingerprint recognition [9], handwritten character recognition [10,11], and medical analysis [12].…”

Section: Introductionmentioning

confidence: 99%

An edge thinning algorithm based on newly defined single‐pixel edge patterns

Ren

Wang

et al. 2022

IET Image Processing

View full text Add to dashboard Cite

To improve the uniformity of one‐pixel width and continuity of the thinned edges, this paper proposes an edge thinning algorithm acting on grey‐scale edge images based on 24 self‐defined single‐pixel connection patterns. First, for binary or blurred grey‐scale gradient edge images, a distance–greyscale coupling algorithm is proposed to achieve gradient enhancement in the edge width direction. Then the elimination rules of noise points and gradient calculation method are given. Secondly, the marking rules of the first three pixels of each edge are given. The next pixel to be marked must meet that the new last three pixels belong to the 24 connection modes. Whether the qualified pixels are retained depends on its grey value and the local edge gradient. The algorithm is tested on four types of images. The results show that the proposed method can guarantee uniform, smooth, and connected one‐pixel‐wide lines that lie at the centre of the initial edges. The algorithm and the existing algorithms are performed on portrait image and four scenarios of the indoor datasets. Five evaluation indicators are statistically analyzed to prove the feasibility and effectiveness of the proposed algorithm.

show abstract

Action recognition based on RGB and skeleton data sets: A survey

Cited by 39 publications

References 104 publications

Multimodal cooperative self‐attention network for action recognition

Multimodal cooperative self‐attention network for action recognition

Distinct Motion-Preserving Graph Convolutional Network for Two-Person Interaction Recognition

An edge thinning algorithm based on newly defined single‐pixel edge patterns

Contact Info

Product

Resources

About