Relaxed Transformer Decoders for Direct Action Proposal Generation

Abstract

Temporal action proposal generation is an important and challenging task in video understanding, which aims at detecting all temporal segments containing action instances of interest. The existing proposal generation approaches are generally based on pre-defined anchor windows or heuristic bottom-up boundary matching strategies. This paper presents a simple and end-to-end learnable framework (RTD-Net) for direct action proposal generation, by re-purposing a Transformer-alike architecture. To tackle the essential visual difference between time and space, we make three important improvements over the original transformer detection framework (DETR). First, to deal with slowness prior in videos, we replace the original Transformer encoder with a boundary attentive module to better capture temporal information. Second, due to the ambiguous temporal boundary and relatively sparse annotations, we present a relaxed matching loss to relieve the strict criteria of single assignment to each groundtruth. Finally, we devise a three-branch head to further improve the proposal confidence estimation by explicitly predicting its completeness. Extensive experiments on THUMOS14 and ActivityNet-1.3 benchmarks demonstrate the effectiveness of RTD-Net, on both tasks of temporal action proposal generation and temporal action detection. Moreover, due to its simplicity in design, our RTD-Net is more efficient than previous proposal generation methods without non-maximum suppression post-processing. The code will be available at https://github.com/MCG-NJU/RTD-Action.

Deep Learning-based Action Detection in Untrimmed Videos: A Survey

“…Among them VSGN [64] achieved the best performance by exploiting correlations between cross-scale snippets (original and magnified) and aggregating their features with a graph pyramid network. AGT [65], RTD-Net [66], ATAG [63],…”

“…and TadTR [152] use transformers to model long-range dependencies. Among them RTD-Net [66] achieved the There are also two state-of-the-art (SOTA) methods that do not belong to the mentioned categories of methods. TSP [154] proposed a novel supervised pretraining paradigm for clip features, and improved the performance of SOTA using features trained with the proposed pretraining strategy.…”

“…Transformer AGT [65], RTD-Net [66] ATAG [63], TadTR [152] + Modeling non-linear temporal structure and inter-proposal relationships for proposal generation. -High parametric complexity.…”

“…the input video) and graph structured query embeddings (latent representations of the action queries).Tan et al in RTD-Net[66] proposed a relaxed transformer to directly generate action proposals without the need to human prior knowledge for careful design of anchor placement or boundary matching mechanisms. The transformer encoder models long-range temporal context and captures inter-proposal relationships from a global view to precisely localize action instances.…”

LocalViT: Bringing Locality to Vision Transformers

“…Thus, transformers are especially good at modelling long-range dependencies between elements of a sequence. Since then, there have been several attempts to adapt transformers towards vision tasks including object detection [2,56], image classification [8,41,52,46,14], segmentation [44], multiple object tracking [37,29], human pose estimation [50,55], point cloud processing [12,54], video processing [10,31,38], image super-resolution [30,49,3], image synthesis [9], etc. An extensive review is out of the scope of this paper.…”