DirecFormer: A Directed Attention in Transformer Approach to Robust Action Recognition

Truong, Thanh-Dat; Bui, Quoc-Huy; Duong, Chi Nhan; Seo, Han‐Seok; Phung, Son Lam; Li, Xin; Luu, Khoa

doi:10.1109/cvpr52688.2022.01940

Cited by 38 publications

(16 citation statements)

References 36 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In order to improve attention efficiency, Ref. [40] proposes a novel directed-attention mechanism to understand human actions in exact order. A trajectory attention block (trajectory attention block) is proposed [41] to enhance the robustness of human action recognition in dynamic scenes which generates a set of specific trajectory markers along the spatial dimension and performs pooling operations along the temporal dimension.…”

Section: Attention Mechanism For Video Understandingmentioning

confidence: 99%

SlowFast Multimodality Compensation Fusion Swin Transformer Networks for RGB-D Action Recognition

Xiao

Ren

et al. 2023

Mathematics

View full text Add to dashboard Cite

RGB-D-based technology combines the advantages of RGB and depth sequences which can effectively recognize human actions in different environments. However, the spatio-temporal information between different modalities is difficult to effectively learn from each other. To enhance the information exchange between different modalities, we introduce a SlowFast multimodality compensation block (SFMCB) which is designed to extract compensation features. Concretely, the SFMCB fuses features from two independent pathways with different frame rates into a single convolutional neural network to achieve performance gains for the model. Furthermore, we explore two fusion schemes to combine the feature from two independent pathways with different frame rates. To facilitate the learning of features from independent multiple pathways, multiple loss functions are utilized for joint optimization. To evaluate the effectiveness of our proposed architecture, we conducted experiments on four challenging datasets: NTU RGB+D 60, NTU RGB+D 120, THU-READ, and PKU-MMD. Experimental results demonstrate the effectiveness of our proposed model, which utilizes the SFMCB mechanism to capture complementary features of multimodal inputs.

show abstract

Section: Attention Mechanism For Video Understandingmentioning

confidence: 99%

SlowFast Multimodality Compensation Fusion Swin Transformer Networks for RGB-D Action Recognition

Xiao

Ren

et al. 2023

Mathematics

View full text Add to dashboard Cite

show abstract

“…Truong et al propose an end-to-end transformer structure, Direcformer [71]. The structure introduces ordinal time learning into the transformer, which helps to understand the chronological order of actions.…”

Section: Transformer Modelsmentioning

confidence: 99%

“…This initial recommendation set is pruned and temporarily extended using optical flow and transudative learning. 58.5% 65.7% ACT [71] 65.7% 69.5% Faster-RCNN + two-stream I3Dconv [72] 73.3% 76.3% YOWO (16-frame) [73] 74.4% 87.2% HIT [116] 83.8% 84.8%…”

Section: Spatiotemporal Action Detectionmentioning

confidence: 99%

Action Recognition and Detection Based on Deep Learning: A Comprehensive Summary

Li,

Liang,

Gan

et al. 2023

CMC

View full text Add to dashboard Cite

Action recognition and detection is an important research topic in computer vision, which can be divided into action recognition and action detection. At present, the distinction between action recognition and action detection is not clear, and the relevant reviews are not comprehensive. Thus, this paper summarized the action recognition and detection methods and datasets based on deep learning to accurately present the research status in this field. Firstly, according to the way that temporal and spatial features are extracted from the model, the commonly used models of action recognition are divided into the two stream models, the temporal models, the spatiotemporal models and the transformer models according to the architecture. And this paper briefly analyzes the characteristics of the four models and introduces the accuracy of various algorithms in common data sets. Then, from the perspective of tasks to be completed, action detection is further divided into temporal action detection and spatiotemporal action detection, and commonly used datasets are introduced. From the perspectives of the twostage method and one-stage method, various algorithms of temporal action detection are reviewed, and the various algorithms of spatiotemporal action detection are summarized in detail. Finally, the relationship between different parts of action recognition and detection is discussed, the difficulties faced by the current research are summarized in detail, and future development was prospected.

show abstract

“…As Vision Transformers brought recent breakthroughs in computer vision, specifically for action recognition tasks, many researchers have adopted them as their model [ 37 , 38 , 39 , 40 ] or combined them with 2D CNN [ 41 ]. For example, Arnab et al [ 37 ] proposed several factorization variants to model spatial and temporal representation effectively inside a transformer encoder.…”

Section: Related Workmentioning

confidence: 99%

Video Action Recognition Using Motion and Multi-View Excitation with Temporal Aggregation

Joefrie

Aono

2022

Entropy

View full text Add to dashboard Cite

Spatiotemporal and motion feature representations are the key to video action recognition. Typical previous approaches are to utilize 3D CNNs to cope with both spatial and temporal features, but they suffer from huge computations. Other approaches are to utilize (1+2)D CNNs to learn spatial and temporal features in an efficient way, but they neglect the importance of motion representations. To overcome problems with previous approaches, we propose a novel block which makes it possible to alleviate the aforementioned problems, since our block can capture spatial and temporal features more faithfully and efficiently learn motion features. This proposed block includes Motion Excitation (ME), Multi-view Excitation (MvE), and Densely Connected Temporal Aggregation (DCTA). The purpose of ME is to encode feature-level frame differences; MvE is designed to enrich spatiotemporal features with multiple view representations adaptively; and DCTA is to model long-range temporal dependencies. We inject the proposed building block, which we refer to as the META block (or simply “META”), into 2D ResNet-50. Through extensive experiments, we demonstrate that our proposed method architecture outperforms previous CNN-based methods in terms of “Val Top-1 %” measure with Something-Something v1 and Jester datasets, while the META yielded competitive results with the Moment-in-Time Mini dataset.

show abstract

DirecFormer: A Directed Attention in Transformer Approach to Robust Action Recognition

Cited by 38 publications

References 36 publications

SlowFast Multimodality Compensation Fusion Swin Transformer Networks for RGB-D Action Recognition

SlowFast Multimodality Compensation Fusion Swin Transformer Networks for RGB-D Action Recognition

Action Recognition and Detection Based on Deep Learning: A Comprehensive Summary

Video Action Recognition Using Motion and Multi-View Excitation with Temporal Aggregation

Contact Info

Product

Resources

About