MMNet: A Model-based Multimodal Network for Human Action Recognition in RGB-D Videos

Yu, Bruce X. B.; Liu, Yan; Zhang, Xiang; Zhong, Sheng-hua; Chan, Keith C. C.

doi:10.1109/tpami.2022.3177813

Cited by 30 publications

(21 citation statements)

References 78 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Year cs(%) cv (%) STA-LSTM [54] 2017 61.20 63.30 JCRRNN [55] 2016 64.60 66.90 Skeleton boxes [56] 2017 82.50 84.90 Li et al [57] 2017 86.80 94.20 HCN [58] 2018 85.90 87.60 TSMF [59] 2021 95.8 97.8 MMNet [60] 2022 97.4 98.6 (Ours) SAt-Object Integration -98.1 98.9…”

Section: Methodsmentioning

confidence: 99%

Object Knowledge Integration for Skeleton-Based Action Recognition

Moutik,

Sekkat,

Ait Tchakoucht

et al. 2024

Stat., optim. inf. comput.

View full text Add to dashboard Cite

3D-Skeleton-based action recognition has been widely adopted due to its efficiency and robustness to complex backgrounds. While it is capable of conveying a significant amount of information regarding the dynamics of human poses, we argue that its performance is curtailed when confronted with actions involving interactions between humans and objects due to the absence of the study of the surrounding objects. It is of great importance to delve deeper into the study of human-object interactions for skeleton-based action recognition. This paper proposes a novel approach to represent the spatial-temporal skeleton features, along with the present nearby objects and their dynamics. To accomplish this, a new formulation named object knowledge is presented, which entails the categorization of object characteristics, based on whether or not the object necessitates a motion analysis. With a piece of prior knowledge, in cases where it is required, the motion is calculated, while for those where it is not necessary, only the category of object is considered. This object knowledge is then early-fusion along with the skeleton representation, in such a way that it fits into the self-attention model. The experimental results on different popular action recognition datasets (NTU RGB+D 60, NTU RGB+ D 120 illustrate that the proposed approach outperforms the current state-of-the-art methods.

show abstract

Section: Methodsmentioning

confidence: 99%

Object Knowledge Integration for Skeleton-Based Action Recognition

Moutik,

Sekkat,

Ait Tchakoucht

et al. 2024

Stat., optim. inf. comput.

View full text Add to dashboard Cite

show abstract

“…Li et al [16] approach to fusion, which is based on models, is distinct from other existing approaches to fusion at the model level that necessitate similarity in representation across multiple modalities and focuses on fusion with co-learning, utilizing a thorough comprehension of the data structure. Specifically, Yu et al [25] acquired representation from the RGB modality by emphasizing body parts that provided mutually complementary qualities to the skeleton modality which may limit the generalizability of the results to other data-sets or real-world scenarios.…”

Section: Multimodal Harmentioning

confidence: 99%

Multi-Modal Human Action Recognition in videos using LSTM with Feature Fusion

Bharathi,

Sri

2024

Preprint

View full text Add to dashboard Cite

Over past few decades, Human action recognition (HAR) in RGB-Depth videos has been extensively studied on the availability of economical depth sensors. Presently, unimodal methods such as skeleton-based and RGB video-based have achieved significant advancements by utilizing progressively larger amounts of data. Nevertheless, there has been a challenge of extracting the temporal feature in RGB video, spatial feature in skeleton sequence. To solve this challenge, we propose Skeleton-RGB Feature Fusion LSTM (SRF-LSTM) network to combine the skeleton and RGB stream modalities and to recognize actions. The fusion of RGB video and skeleton sequences capture both temporal and spatial information effectively. The proposed SRF-LSTM model performs better with an average accuracy of 90.4% than state-of-the-art techniques on NTU RGB-D dataset.

show abstract

“…We employ view1 and view2 as the training set, and view3 as the validation set on N-UCLA dataset. There are three parts in this table: knowledge distillation methods [10], [24], [25], multi-modality fusion methods [4], [46], [47], [48], [49], and the proposed methods. Our FCKD(F+S+D) achieves 98.3% accuracy, outperforming the baseline 3D ResNeXt101 by 3.7%.…”

Section: E Comparison With State-of-the-artsmentioning

confidence: 99%

“…These results demonstrate that our method can learn view-invariant semantics by focusing on the focal channel features between modalities, improving the accuracy on the crossview N-UCLA dataset. Compared to multi-modality fusion methods [4], [46], [47], [48], [49], our FCKD(F+S+D) only uses the RGB modality in the testing phase, but obtains a superior accuracy to Hybrid [46], VPN [47], MMNet [48] and 3DV [4] by 5.2%, 4.8%, 4.6% and 3.0%. When we fuse FCKD(F+S+D) with unimodal optical flow, skeleton and depth, our FCKD(F+S+D)+F+S+D obtains the state-of-the-art accuracy of 98.9% which is comparable to Hierarchical [49].…”

Section: E Comparison With State-of-the-artsmentioning

confidence: 99%

Focal Channel Knowledge Distillation for Multi-Modality Action Recognition

Gan

Cao

et al. 2023

IEEE Access

View full text Add to dashboard Cite

The multi-modality action recognition aims to learn the complementary information from multiple modalities to improve the action recognition performance. However, there exists a significant modality channel difference, the equal transferring channel semantic features from multi-modalities to RGB will result in competition and redundancy during knowledge distillation. To address this issue, we propose a focal channel knowledge distillation strategy to transfer the key semantic correlations and distributions of multi-modality teachers into the RGB student network. The focal channel correlations provide intrinsic relationships and diversity properties of key semantics, and focal channel distributions provide salient channel activation of features. By ignoring the less-discriminative and irrelevant channels, the student can more efficiently utilize the channel capability to learn the complementary semantic features from the other modalities. Our focal channel knowledge distillation achieves 91.2%, 95.6%, 98.3% and 81.0% accuracy with 4.5%, 4.2%, 3.7% and 7.1% improvement on NTU 60 (CS), UTD-MHAD, N-UCLA and HMDB51 datasets comparing to unimodal RGB models. This focal channel knowledge distillation framework can also be integrated with the unimodal models to achieve the state-of-the-art performance. The extensive experiments show that the proposed method achieves 92.5%, 96.0%, 98.9%, and 82.3% accuracy on NTU 60 (CS), UTD-MHAD, N-UCLA, and HMDB51 datasets respectively.

show abstract

MMNet: A Model-based Multimodal Network for Human Action Recognition in RGB-D Videos

Cited by 30 publications

References 78 publications

Object Knowledge Integration for Skeleton-Based Action Recognition

Object Knowledge Integration for Skeleton-Based Action Recognition

Multi-Modal Human Action Recognition in videos using LSTM with Feature Fusion

Focal Channel Knowledge Distillation for Multi-Modality Action Recognition

Contact Info

Product

Resources

About