End-to-end Temporal Action Detection with Transformer

Abstract

Temporal action detection (TAD) aims to determine the semantic label and the boundaries of every action instance in an untrimmed video. It is a fundamental task in video understanding and significant progress has been made in TAD. Previous methods involve multiple stages or networks and hand-designed rules or operations, which fall short in efficiency and flexibility. Here, we construct an end-toend framework for TAD upon Transformer, termed TadTR, which simultaneously predicts all action instances as a set of labels and temporal locations in parallel. TadTR is able to adaptively extract temporal context information needed for making action predictions, by selectively attending to a number of snippets in a video. It greatly simplifies the pipeline of TAD and runs much faster than previous detectors. Our method achieves state-of-the-art performance on HACS Segments and THUMOS14 and competitive performance on ActivityNet-1.3. Our code will be made available at https://github.com/xlliu7/TadTR.

OWL (Observe, Watch, Listen): Localizing Actions in Egocentric Video via Audiovisual Temporal Context

“…[46,47] propose graph-based methods, where they define proposals and snippets as graph nodes and perform graph convolutions for the information exchange. Our approach is closer to recent work that leverage the Transformer architecture [30,33,38]. Due to the rising popularity of transformers for vision tasks [3,10,16], [30,33,38] extended the transformer building blocks to the inner working of TAL as a way to infuse temporal context between proposals.…”

“…Our approach is closer to recent work that leverage the Transformer architecture [30,33,38]. Due to the rising popularity of transformers for vision tasks [3,10,16], [30,33,38] extended the transformer building blocks to the inner working of TAL as a way to infuse temporal context between proposals. In contrast to prior art, our work considers the interplay of multiple modalities, visual and audio, while also modeling the surrounding context of an action.…”

Deep Learning-based Action Detection in Untrimmed Videos: A Survey

“…and TadTR [152] use transformers to model long-range dependencies. Among them RTD-Net [66] achieved the There are also two state-of-the-art (SOTA) methods that do not belong to the mentioned categories of methods.…”

“…Transformer AGT [65], RTD-Net [66] ATAG [63], TadTR [152] + Modeling non-linear temporal structure and inter-proposal relationships for proposal generation. -High parametric complexity.…”

PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer

“…Many other ViT variants [8,13,21,22,25,37,54,60,70] are proposed from then, which achieve promising performance compared with its counterpart CNNs for image analysis tasks [6,23,74]. Recently, some works introduce vision transformer for video understanding tasks such as action recognition [1,3,4,15,20,38,42], action detection [36,58,62,73], video superresolution [5], video inpainting [32,71], and 3D animation [9]. Some works [20,42] conduct temporal contextual modeling with transformer based on single-frame features from pretrained 2D networks, while other works [1,3,4,15,38] mine the spatio-temporal attentions via video transformer directly.…”