Multiview Transformers for Video Recognition

Shen, Yan; Xiong, Xuehan; Arnab, Anurag; Lu, Zhichao; Zhang, Mi; Sun, Cheng; Schmid, Cordelia

doi:10.48550/arxiv.2201.04288

Cited by 3 publications

(6 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Attention-based video methods [58,60] have initially been used as part of spatio-temporal CNN architectures [6,56]. The recent introduction of Vision Transformer [10], has inspired subsequent works on action recognition by either focusing on how spatio-temporal information can be processed [1,2] or architectural optimizations for spatio-temporal data [12,39,47,66,68]. Motivated by the recent advances of spatio-temporal transformers for action recognition, we combine multiple transformer towers in our TemPr model.…”

Section: Related Workmentioning

confidence: 99%

Temporal Progressive Attention for Early Action Prediction

Stergiou¹,

Damen²

2022

Preprint

View full text Add to dashboard Cite

Early action prediction deals with inferring the ongoing action from partially-observed videos, typically at the outset of the video. We propose a bottleneck-based attention model that captures the evolution of the action, through progressive sampling over fine-to-coarse scales. Our proposed Temporal Progressive (TemPr) model is composed of multiple attention towers, one for each scale. The predicted action label is based on the collective agreement considering confidences of these attention towers. Extensive experiments over three video datasets showcase state-of-the-art performance on the task of Early Action Prediction across a range of backbone architectures. We demonstrate the effectiveness and consistency of TemPr through detailed ablations. 1

show abstract

Section: Related Workmentioning

confidence: 99%

Temporal Progressive Attention for Early Action Prediction

Stergiou¹,

Damen²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Uniformer [28] is a custom fused CNN-Transformer architecture achieving good speed-accuracy trade-off. Yan et al [52] propose a multi-stream Transformer operating on different resolutions with lateral connections. Prior work [5,3,52] has shown the benefit of image pretraining for Class Score…”

Section: Related Workmentioning

confidence: 99%

“…Yan et al [52] propose a multi-stream Transformer operating on different resolutions with lateral connections. Prior work [5,3,52] has shown the benefit of image pretraining for Class Score…”

Section: Related Workmentioning

confidence: 99%

“…As a fundamental component of video understanding, learning spatiotemporal representations remains an active research area in recent years. Since the beginning of the deep learning era, numerous architectures have been proposed to learn spatiotemporal semantics, such as traditional two-stream networks [40,46,59], 3D convolutional neural networks [42,5,20,35,44,50,48,13,12], and spatiotemporal Transformers [3,33,32,10,1,28,52]. As videos are high-dimensional and exhibit substantial spatiotemporal redundancy, training video recognition models from scratch is highly inefficient and may lead to inferior performance.…”

Section: Introductionmentioning

confidence: 99%

“…Intuitively, the semantic meaning of a video snippet is highly correlated with each of its individual frames. Previous studies [5,3,1,52] have shown that the datasets and methodologies for image recognition can benefit video recognition as well. Owing to the close relationship between image and video recognition, as a routine practice, most existing video recognition models take advantage of pretrained image models by using them for initialization and then re-training all parameters for video understanding in an end-to-end manner.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Frozen CLIP Models are Efficient Video Learners

Lin¹,

Geng²,

Zhang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Video recognition has been dominated by the end-to-end learning paradigm -first initializing a video recognition model with weights of a pretrained image model and then conducting end-to-end training on videos. This enables the video network to benefit from the pretrained image model. However, this requires substantial computation and memory resources for finetuning on videos and the alternative of directly using pretrained image features without finetuning the image backbone leads to subpar results. Fortunately, recent advances in Contrastive Vision-Language Pre-training (CLIP) pave the way for a new route for visual recognition tasks. Pretrained on large open-vocabulary image-text pair data, these models learn powerful visual representations with rich semantics. In this paper, we present Efficient Video Learning (EVL) -an efficient framework for directly training high-quality video recognition models with frozen CLIP features. Specifically, we employ a lightweight Transformer decoder and learn a query token to dynamically collect frame-level spatial features from the CLIP image encoder. Furthermore, we adopt a local temporal module in each decoder layer to discover temporal clues from adjacent frames and their attention maps. We show that despite being efficient to train with a frozen backbone, our models learn high quality video representations on a variety of video recognition datasets. Code is available at https://github.com/OpenGVLab/efficient-video-recognition.

show abstract

Active Token Mixer

Wei

Zhang

Lan

et al. 2023

AAAI

View full text Add to dashboard Cite

The three existing dominant network families, i.e., CNNs, Transformers and MLPs, differ from each other mainly in the ways of fusing spatial contextual information, leaving designing more effective token-mixing mechanisms at the core of backbone architecture development. In this work, we propose an innovative token-mixer, dubbed Active Token Mixer (ATM), to actively incorporate contextual information from other tokens in the global scope into the given query token. This fundamental operator actively predicts where to capture useful contexts and learns how to fuse the captured contexts with the query token at channel level. In this way, the spatial range of token-mixing can be expanded to a global scope with limited computational complexity, where the way of token-mixing is reformed. We take ATMs as the primary operators and assemble them into a cascade architecture, dubbed ATMNet. Extensive experiments demonstrate that ATMNet is generally applicable and comprehensively surpasses different families of SOTA vision backbones by a clear margin on a broad range of vision tasks, including visual recognition and dense prediction tasks. Code is available at https://github.com/microsoft/ActiveMLP.

show abstract

Multiview Transformers for Video Recognition

Cited by 3 publications

References 40 publications

Temporal Progressive Attention for Early Action Prediction

Temporal Progressive Attention for Early Action Prediction

Frozen CLIP Models are Efficient Video Learners

Active Token Mixer

Contact Info

Product

Resources

About