SlowFast Multimodality Compensation Fusion Swin Transformer Networks for RGB-D Action Recognition

Abstract

RGB-D-based technology combines the advantages of RGB and depth sequences which can effectively recognize human actions in different environments. However, the spatio-temporal information between different modalities is difficult to effectively learn from each other. To enhance the information exchange between different modalities, we introduce a SlowFast multimodality compensation block (SFMCB) which is designed to extract compensation features. Concretely, the SFMCB fuses features from two independent pathways with different frame rates into a single convolutional neural network to achieve performance gains for the model. Furthermore, we explore two fusion schemes to combine the feature from two independent pathways with different frame rates. To facilitate the learning of features from independent multiple pathways, multiple loss functions are utilized for joint optimization. To evaluate the effectiveness of our proposed architecture, we conducted experiments on four challenging datasets: NTU RGB+D 60, NTU RGB+D 120, THU-READ, and PKU-MMD. Experimental results demonstrate the effectiveness of our proposed model, which utilizes the SFMCB mechanism to capture complementary features of multimodal inputs.

Comparison Analysis of Multimodal Fusion for Dangerous Action Recognition in Railway Construction Sites

“…UTKinect-Action3D [63], CAD-60 [64,65], LIRIS human activities Product score fusion (late fusion) [34] Visual (RGB) and depth dynamic images c-ConvNet ChaLearn LAP IsoGD [61], NTU RGB+D [53] Fusion score (late fusion) [33] RGB and depth frames Pre-trained VGG networks MSRDaily activity 3D [49], UTD-MHAD [51], CAD-60 [64,65] early, intermediate, and late fusion [13] RGB, depth, skeleton I3D and shift-GCN NTU RGB+D [53], SBU interaction [66] Cross-modality compensation block (intermediate) [42] RGB and depth ResNet and VGG with CMCB NTU RGB+D 120 [56], THU-READ [58], PKU-MMD [67] SlowFast multimodality compensation block (intermediate) [40] RGB & Depth features Swin transformer NTU RGB+D 120 [56], NTU RGB+D 60 [53], THU-READ [58], PKU-MMD [67] Cross-modality fusion transformer (intermediate) [41] RGB and depth features Restnet50 feature extractors NTU RGB+D 120 [56], THU-READ [58], PKU-MMD [67]…”

“…The authors in [39] propose a Transformer-based framework for egocentric action recognition, emphasizing inter-frame and mutual-attentional fusion. The authors in [40] contrast previous methods by integrating a SlowFast multimodality compensation block for processing depth and RGB sequences. The authors in [41] propose a dual-stream cross-modality fusion transformer, enhancing feature representations through self-attention and cross-attention mechanisms.…”

Video-Audio Domain Generalization via Confounder Disentanglement

“…• SlowFast (SF) (Xiao et al 2020;Feichtenhofer et al 2019) is a publicly available video-audio action recognition baseline.…”

“…Many existing works focus on mining domain-invariant features and thus blocking the impact of domain-variant features that hinder the generalization across domains. However, they are vulnerable to (Xiao et al 2020;Feichtenhofer et al 2019) in intra-domain and DG settings with Advertisement, Cartoon, Game, and Movie domains. r and e denote train and test sets.…”