Context-Aware Cross-Attention for Skeleton-Based Human Action Recognition

Fan, Yanbo; Weng, Shuchen; Zhang, Yong; Shi, Boxin; Zhang, Yi

doi:10.1109/access.2020.2968054

Cited by 34 publications

(18 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…On NTU-RGB+D dataset, the model achieves the latest performance, which verifies the effectiveness of the model in behavior recognition tasks. DSSCA-SSLM [32] 74.9% TCN [33] 74.3% 83.10% GCA-LSTM [16] 76.1% 84.0% Skelemotion [34] 76.5% 84.7% Slowfastnet [12] 80.25% 93.74% St-gcn [18] 81.5% 88.3% LSTM-CNN [35] 82.9% 91.0% Two-stream CNN [36] 83.2% 89.3% DPRL+GCNN [19] 83.5% 89.8% Cross-attention [21] 84.2% 89.3% SV-GCN (ours) 85.51% 94.15% 9 Journal of Sensors…”

Section: Discussionmentioning

confidence: 99%

“…Action recognition methods based on multiple data sources, Fan et al [21] proposed context-aware crossattention for skeleton-based human action recognition, proposed a cross-attention module that can extract context information directly from original RGB video, and used it in action recognition methods based on skeleton data. However, this method can only use scene context information in the original video and cannot fully use all information in the original video and does not use GCN-based method to model skeleton data.…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

RGB‐D Human Action Recognition of Deep Feature Enhancement and Fusion Using Two‐Stream ConvNet

Liu

et al. 2021

Journal of Sensors

View full text Add to dashboard Cite

Action recognition is an important research direction of computer vision, whose performance based on video images is easily affected by factors such as background and light, while deep video images can better reduce interference and improve recognition accuracy. Therefore, this paper makes full use of video and deep skeleton data and proposes an RGB-D action recognition based two-stream network (SV-GCN), which can be described as a two-stream architecture that works with two different data. Proposed Nonlocal-stgcn (S-Stream) based on skeleton data, by adding nonlocal to obtain dependency relationship between a wider range of joints, to provide more rich skeleton point features for the model, proposed a video based Dilated-slowfastnet (V-Stream), which replaces traditional random sampling layer with dilated convolutional layers, which can make better use of depth the feature; finally, two stream information is fused to realize action recognition. The experimental results on NTU-RGB+D dataset show that proposed method significantly improves recognition accuracy and is superior to st-gcn and Slowfastnet in both CS and CV.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

RGB‐D Human Action Recognition of Deep Feature Enhancement and Fusion Using Two‐Stream ConvNet

Liu

et al. 2021

Journal of Sensors

View full text Add to dashboard Cite

show abstract

“…Therefore, they capture deep contextualized information of the input sequence [18]. Fan et al [19] proposed a self-attention module, which helps to extract joint features with more information and highly related to the context information of the corresponding scene, so as to suppress the influence of irrelevant joint features. The experimental results show that the accuracy is 98.59% adding self-attention mechanism.…”

Section: Deep Learning Of Action Feature Presentation Based On Attention Mechanismmentioning

confidence: 99%

Survey of Action Recognition Based on Attention Mechanism

Yao

et al. 2021

Proceedings of 2021 Chinese Intelligent Automation Conference

View full text Add to dashboard Cite

Attention mechanism for action recognition is to enhance data by using attention mechanism so as to recognize human action in video. This research has been intensely investigated in the field of computer vision, which has attracted extensive attention from academia and industry. It is also the premise of intelligent interaction and human-computer cooperation, which helps the machine to perceive the external environment. In the past decade, especially the emergence of deep learning technology, great progress has been made in this field. Therefore, it is necessary to make a comprehensive review of the recent work. This paper summarizes the research background and application fields, typical datasets and experimental evaluation, and compares various attention mechanism for human action feature representation and action classification algorithms. Finally, we look forward to the development prospect of human action recognition in self-driving vehicles. This paper provides ideas for the follow-up research on the understanding of the behavior of self-driving vehicles, and makes self-driving vehicles become interactive wheeled robots.

show abstract

“…Human-object interaction is a very popular research area due to its wide applicability to, e.g., healthcare monitoring [9], assisted living [10], surveillance [11], motion sensing games [12], content-based video indexing and retrieval, etc., in the field of computer vision [13,14]. Thus, there is a need for a more reliable and accurate system.…”

Section: Introductionmentioning

confidence: 99%

Semantic Recognition of Human-Object Interactions via Gaussian-Based Elliptical Modeling and Pixel-Level Labeling

et al. 2021

View full text Add to dashboard Cite

Human-Object Interaction (HOI) recognition, due to its significance in many computer visionbased applications, requires in-depth and meaningful details from image sequences. Incorporating semantics in scene understanding has led to a deep understanding of human-centric actions. Therefore, in this research work, we propose a semantic HOI recognition system based on multi-vision sensors. In the proposed system, the de-noised RGB and depth images, via Bilateral Filtering (BLF), are segmented into multiple clusters using a Simple Linear Iterative Clustering (SLIC) algorithm. The skeleton is then extracted from segmented RGB and depth images via Euclidean Distance Transform (EDT). Human joints, extracted from the skeleton, provide the annotations for accurate pixel-level labeling. An elliptical human model is then generated via a Gaussian Mixture Model (GMM). A Conditional Random Field (CRF) model is trained to allocate a specific label to each pixel of different human body parts and an interaction object. Two semantic feature types that are extracted from each labeled body part of the human and labelled objects are: Fiducial points and 3D point cloud. Features descriptors are quantized using Fisher's Linear Discriminant Analysis (FLDA) and classified using K-ary Tree Hashing (KATH). In experimentation phase the recognition accuracy achieved with the Sports dataset is 92.88%, with the Sun Yat-Sen University (SYSU) 3D HOI dataset is 93.5% and with the Nanyang Technological University (NTU) RGB+D dataset it is 94.16%. The proposed system is validated via extensive experimentation and should be applicable to many computer-vision based applications such as healthcare monitoring, security systems and assisted living etc.INDEX TERMS 3D point cloud, fiducial points, human-object interaction, pixel labeling, semantic segmentation, super-pixels, K-ary tree hashing.

show abstract

Context-Aware Cross-Attention for Skeleton-Based Human Action Recognition

Cited by 34 publications

References 31 publications

RGB‐D Human Action Recognition of Deep Feature Enhancement and Fusion Using Two‐Stream ConvNet

RGB‐D Human Action Recognition of Deep Feature Enhancement and Fusion Using Two‐Stream ConvNet

Survey of Action Recognition Based on Attention Mechanism

Semantic Recognition of Human-Object Interactions via Gaussian-Based Elliptical Modeling and Pixel-Level Labeling

Contact Info

Product

Resources

About