Deep Concept-wise Temporal Convolutional Networks for Action Localization

Li, Xin; Lin, Tianwei; Liu, Xiao; Zuo, Wangmeng; Li, Chao; Long, Xiang; He, Dongliang; Fu, Li; Wen, Shifeng; Gan, Chuang

doi:10.1145/3394171.3413860

Cited by 28 publications

(8 citation statements)

References 96 publications

(186 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our method achieves an average mAP of 62.7% ([0.3 : 0.1 : 0.7]), with an mAP of 65.6% at tIoU=0.5 and an mAP of 42.6% at tIoU=0.7, outperforming all previous methods by a large margin (+8.7% mAP at tIoU=0.5 and +11.6% mAP at tIoU=0.7). Our results stay on top of all single-stage methods, and also beats all previous two-stage methods, including the latest ones from [33,51,58,84]. Note that our method significantly outperforms the concurrent work of TadTR [46], which also designed a Transformer model for TAL.…”

Section: Resultssupporting

confidence: 58%

ActionFormer: Localizing Moments of Actions with Transformers

Zhang¹,

Wu²,

Li³

2022

Preprint

View full text Add to dashboard Cite

Self-attention based Transformer models have demonstrated impressive results for image classification and object detection, and more recently for video understanding. Inspired by this success, we investigate the application of Transformer networks for temporal action localization in videos. To this end, we present ActionFormer-a simple yet powerful model to identify actions in time and recognize their categories in a single shot, without using action proposals or relying on pre-defined anchor windows. ActionFormer combines a multiscale feature representation with local selfattention, and uses a light-weighted decoder to classify every moment in time and estimate the corresponding action boundaries. We show that this orchestrated design results in major improvements upon prior works. Without bells and whistles, ActionFormer achieves 65.6% mAP at tIoU=0.5 on THUMOS14, outperforming the best prior model by 8.7 absolute percentage points and crossing the 60% mAP for the first time. Further, ActionFormer demonstrates strong results on ActivityNet 1.3 (36.0% average mAP) and the more recent EPIC-Kitchens 100 (+13.5% average mAP over prior works). Our code is available at https://github. com/happyharrycn/actionformer_release.

show abstract

Section: Resultssupporting

confidence: 58%

ActionFormer: Localizing Moments of Actions with Transformers

Zhang¹,

Wu²,

Li³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Depending on temporal level variations, for example, C-TCN [18] proposes Random Move to simulate the montage effect of video production. However, image level variations have been neglected for a long time by the TAD community so that most of previous works don't realize the effectiveness of image level data augmentation (ILDA) for TAD model training.…”

Section: Image Level Data Augmentationmentioning

confidence: 99%

RGB Stream Is Enough for Temporal Action Detection

Wang,

Cai,

Zou

et al. 2021

Preprint

View full text Add to dashboard Cite

State-of-the-art temporal action detectors to date are based on two-stream input including RGB frames and optical flow. Although combining RGB frames and optical flow boosts performance significantly, optical flow is a handdesigned representation which not only requires heavy computation, but also makes it methodologically unsatisfactory that two-stream methods are often not learned end-to-end jointly with the flow. In this paper, we argue that optical flow is dispensable in high-accuracy temporal action detection and image level data augmentation (ILDA) is the key solution to avoid performance degradation when optical flow is removed. To evaluate the effectiveness of ILDA, we design a simple yet efficient one-stage temporal action detector based on single RGB stream named DaoTAD. Our results show that when trained with ILDA, DaoTAD has comparable accuracy with all existing state-of-the-art two-stream detectors while surpassing the inference speed of previous methods by a large margin and the inference speed is astounding 6668 fps on GeForce GTX 1080 Ti.Code is available at https://github.com/Media-Smart/vedatad.

show abstract

“…[40] found that fusing RGB and Optical Flow streams at the last convolutional layer yields good visual modality features. The resulting mid-level features have been successfully employed by well performing TAL approaches [41,42,43,44]. In particular, they are utilized by G-TAD [2] to obtain feature representations for each temporal proposal.…”

Section: Related Workmentioning

confidence: 99%

Hear Me Out: Fusional Approaches for Audio Augmented Temporal Action Localization

Bagchi¹,

Mahmood²,

Fernandes³

et al. 2021

Preprint

View full text Add to dashboard Cite

State of the art architectures for untrimmed video Temporal Action Localization (TAL) have only considered RGB and Flow modalities, leaving the information-rich audio modality totally unexploited. Audio fusion has been explored for the related but arguably easier problem of trimmed (clip-level) action recognition. However, TAL poses a unique set of challenges. In this paper, we propose simple but effective fusion-based approaches for TAL. To the best of our knowledge, our work is the first to jointly consider audio and video modalities for supervised TAL. We experimentally show that our schemes consistently improve performance for state of the art video-only TAL approaches. Specifically, they help achieve new state of the art performance on largescale benchmark datasets -

show abstract

Deep Concept-wise Temporal Convolutional Networks for Action Localization

Cited by 28 publications

References 96 publications

ActionFormer: Localizing Moments of Actions with Transformers

ActionFormer: Localizing Moments of Actions with Transformers

RGB Stream Is Enough for Temporal Action Detection

Hear Me Out: Fusional Approaches for Audio Augmented Temporal Action Localization

Contact Info

Product

Resources

About