Spatio-Temporal Adaptive Network With Bidirectional Temporal Difference for Action Recognition

Li, Zhilei; Li, Jun; Ma, Yuqing; Wang, Rui; Shi, Zhiping; Ding, Yifu; Liu, Xianglong

doi:10.1109/tcsvt.2023.3250646

Cited by 19 publications

(4 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, as discussed in [22], the spatial information and the temporal information should be learned by different cognitive mechanisms, and should be synchronized to process the sequential information. The conventional approaches [41], [45], [46] that first extract spatial information from each frame and then fuse it with temporal transformers may lead to a lack of information synchronization. Besides, in group detection, the temporal information e.g., trajectories, is not just a sequence, it is also the important location information for distinguishing the group results.…”

Section: Spatio-temporal Methodsmentioning

confidence: 99%

Toward Grouping in Large Scenes With Occlusion-Aware Spatio–Temporal Transformers

Zhang,

Gu,

Lai

et al. 2024

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

Group detection, especially for large-scale scenes, has many potential applications for public safety and smart cities. Existing methods fail to cope with frequent occlusions in large-scale scenes with multiple persons, and are difficult to effectively utilize spatio-temporal information. In this paper, we propose an end-to-end framework, GroupTransformer, for group detection in large-scale scenes. To deal with the frequent occlusions caused by multiple people, we design an occlusion encoder to detect and suppress severely occluded person crops. To explore the potential spatio-temporal relationship, we propose spatio-temporal transformers to simultaneously extract trajectory information and fuse inter-person features in a hierarchical manner. Experimental results on both large-scale and small-scale scenes demonstrate that our method achieves better performance compared with state-of-the-art methods. On large-scale scenes, our method significantly boosts the performance in terms of precision and F1 score by more than 10%. On small-scale scenes, our method still improves the performance of F1 score by more than 5%. We will release the code for research purposes.

show abstract

Section: Spatio-temporal Methodsmentioning

confidence: 99%

Toward Grouping in Large Scenes With Occlusion-Aware Spatio–Temporal Transformers

Zhang,

Gu,

Lai

et al. 2024

IEEE Trans. Circuits Syst. Video Technol.

View full text Add to dashboard Cite

show abstract

“…The comparison with the state-of-the-art methods on the HMDB51 dataset is shown in Table 6, and we report the mean class accuracy over three splits on the HMDB51 dataset. There are three parts in this table: single modality methods [7], [8], [34], [35], [36], multi-stream fusion methods [8], [37], [38], [39], and the proposed methods. Our FCKD(F) achieves an average accuracy of 81.0% on three common splits, outperforming the baseline 3D ResNeXt101 by 7.1%.…”

Section: E Comparison With State-of-the-artsmentioning

confidence: 99%

“…Our FCKD(F) achieves an average accuracy of 81.0% on three common splits, outperforming the baseline 3D ResNeXt101 by 7.1%. Compared with TSM [34], TCM-R50 [35] and STANet [36] that only use RGB modality for temporal modeling, our FCKD(F+S+D) have learned motion knowledge from optical flow and outperforming them by 7.5%, 3.5% and 3.3%, respectively. In addition, the state-of-the-art knowledge distillation methods Mars [8] and D3D [7] also achieve better results than TSM, TCM-R50 and STANet.…”

Section: E Comparison With State-of-the-artsmentioning

confidence: 99%

Focal Channel Knowledge Distillation for Multi-Modality Action Recognition

Gan

Cao

et al. 2023

IEEE Access

View full text Add to dashboard Cite

The multi-modality action recognition aims to learn the complementary information from multiple modalities to improve the action recognition performance. However, there exists a significant modality channel difference, the equal transferring channel semantic features from multi-modalities to RGB will result in competition and redundancy during knowledge distillation. To address this issue, we propose a focal channel knowledge distillation strategy to transfer the key semantic correlations and distributions of multi-modality teachers into the RGB student network. The focal channel correlations provide intrinsic relationships and diversity properties of key semantics, and focal channel distributions provide salient channel activation of features. By ignoring the less-discriminative and irrelevant channels, the student can more efficiently utilize the channel capability to learn the complementary semantic features from the other modalities. Our focal channel knowledge distillation achieves 91.2%, 95.6%, 98.3% and 81.0% accuracy with 4.5%, 4.2%, 3.7% and 7.1% improvement on NTU 60 (CS), UTD-MHAD, N-UCLA and HMDB51 datasets comparing to unimodal RGB models. This focal channel knowledge distillation framework can also be integrated with the unimodal models to achieve the state-of-the-art performance. The extensive experiments show that the proposed method achieves 92.5%, 96.0%, 98.9%, and 82.3% accuracy on NTU 60 (CS), UTD-MHAD, N-UCLA, and HMDB51 datasets respectively.

show abstract

“…TSM [9] innovatively introduces temporal information into 2D CNNs by moving channels across the time dimension, without additional computational overhead. Subsequent studies [10][11][12][13][14][15] integrate various forms of temporal modeling into 2D CNNs with distinct approaches. MVFNet utilizes separable convolutions for dynamic video features but overlooks the correlation among multiscale spatiotemporal features.…”

Section: Introductionmentioning

confidence: 99%

Efficient Spatio-Temporal Network for Action Recognition

Su,

Zhao

2024

Preprint

View full text Add to dashboard Cite

The input tensor of video data includes temporal, spatial, and channel dimensions, crucial for extracting complementary spatial, temporal, and spatio-temporal features for video action recognition. To efficiently extract and integrate these features, we propose an Efficient Spatio-Temporal Module (ESTM) with three pathways dedicated to extracting spatial, temporal, and spatio-temporal features. Each pathway uses the Cross Global Average Pooling (CGAP) module to compress the current dimension, focusing features on the remaining two dimensions. This enhances feature extraction and recognition rates for complex actions. We also introduce a Motion Excitation Module (MEM) to enrich input features by transforming correlations between adjacent frames, reducing computational complexity. Finally, ESTM and MEM are seamlessly integrated into a 2D CNN, forming the Efficient Spatio-Temporal Network (ESTN), with minimal impact on network parameters and computational costs. Extensive experiments show that ESTN outperforms state-of-the-art methods on datasets like Something V1 & V2 and HMDB51, validating its effectiveness.

show abstract

Spatio-Temporal Adaptive Network With Bidirectional Temporal Difference for Action Recognition

Cited by 19 publications

References 53 publications

Toward Grouping in Large Scenes With Occlusion-Aware Spatio–Temporal Transformers

Toward Grouping in Large Scenes With Occlusion-Aware Spatio–Temporal Transformers

Focal Channel Knowledge Distillation for Multi-Modality Action Recognition

Efficient Spatio-Temporal Network for Action Recognition

Contact Info

Product

Resources

About