Spatial-Temporal Information Aggregation and Cross-Modality Interactive Learning for RGB-D-Based Human Action Recognition

Abstract

The RGB-D-based human action recognition is gaining increasing attention because the different modalities can provide complementary information. However, the recognition performance is still not satisfactory due to the limited ability to learn spatial-temporal feature and insufficient intermodel interaction. In this paper, we propose a novel approach for RGB-D human action recognition by aggregating spatial-temporal information and implementing cross-modality interactive learning. Firstly, a spatial-temporal information aggregation module (STIAM) is proposed to utilizes sample convolutional neural networks (CNNs) to aggregate the spatial-temporal information in entire RGB-D sequence into lightweight representations efficiently. This allows the model to extract richer spatial-temporal features with limited extra memory and computational cost. Secondly, a cross-modality interactive module (CMIM) is proposed to fully fuse the multi-modal complementary information. Moreover, a multi-modal interactive network (MMINet) is constructed for RGB-D-based action recognition by embeding the above two modules into the two-stream CNNs. In order to verify the universality of our approach, two backbones are deployed in the two-stream architecture, successively. Ablation experiments demonstrate that the proposed STIAM can bring significant improvement in recognizing actions. CMIM can further play the advantages of complementary features of multiple modalities. Extensive experiments on NTU RGB+D 60, NTU RGB+D 120 and PKU-MMD datasets proved the effectiveness of the proposed approach. The proposed approach achieves an accuracy of 94.3% and 96.5% for cross-subject and cross-view on NTU RGB+D 60, 91.7% and 92.6% for cross-subject and cross-setup on NTU RGB+D 120, 93.6% and 94.2% for cross-subject and cross-view on PKU-MMD datasets, which are the state-of-the-art performance. Further analysis denotes that our approach has advantages in recognizing subtle actions.

Multi-Modal Multi Feature Assisted Human Action Recognition through Deep Learning

“…Qin Cheng et al [32] considered RGB and Depth data modalities and proposed a Spatio-temporal information aggregation module (SITAM) which utilizes CNNs to acquire Spatio-temporal information from input data.…”

“…Moreover, they were not provided local motion attributes to the HAR system which results in more misclassification at the action with similar movements. Even though Q. Cheng et al [32] adapted for temporal attention model along with cross model learning, they didn't reveal the inherent characteristics of actions with different speeds and time periods. Further, we can see that the single data model based HARs have gained very less recognition performance.…”

DVANet: Disentangling View and Action Features for Multi-View Action Recognition

“…Additionally, the compatibility of graph convolutional methods with skeleton-based action recognition led to widespread use in the literature (Chi et al 2022;Song et al 2022;Cheng et al 2020). RGBbased action recognition is more sparse in the literature due to its lack of 3D structure, usually only being used in addition to other modalities (Wang et al 2019;Cheng et al 2022;Das et al 2020). Contrary to these previous works, we demonstrate how our RGB-based model is able to achieve state-of-the-art multi-view action recognition performance, even over skeletal-based models.…”

“…Learning view-invariant representations for multi-view action recognition has also been explored before for both uniand multi-modal approaches (Li et al 2018b;Das and Ryoo 2023;Ji et al 2021;Bian et al 2023), and by using either solely convolutional or transformer-based architectures (Cheng et al 2022;Vyas, Rawat, and Shah 2020;Ji et al 2021). Our approach uses a hybrid architecture to limit each of these respective shortcomings, while also novelly introducing a specific configuration of queries for transformer decoders that facilitates feature disentanglement.…”

Multimodal vision-based human action recognition using deep learning: a review

“…New fusion approaches are proposed in a group of studies (Cheng et al 2021;Zhou et al 2021;Tian et al 2020;Wang et al 2019a;Hampiholi et al 2023;Lee et al 2023;Cheng et al 2022). In Cheng et al (2021), a cross-modality compensation block (CMCB) is developed to learn the cross-modality complementary features from RGB and depth modalities.…”