Multiscale Vision Transformers

Fan, Haoqi; Xiong, Bo; Mangalam, Karttikeya; Li, Yanghao; Yan, Zhicheng; Malik, Jitendra; Feichtenhofer, Christoph

doi:10.48550/arxiv.2104.11227

Cited by 69 publications

(123 citation statements)

References 104 publications

Supporting

Mentioning

123

Contrasting

Order By: Relevance

“…Along this line of research, the main focus is to improve the attention mechanism, so that it can satisfy the intrinsic properties of visual signals. For example, MSViT (Fan et al 2021) builds hierarchical attention layers to obtain multi-scale features. Swin Transformers (Liu et al 2021b) introduces a locality constrain into its attention mechanism.…”

Section: Related Work Attention and Vision Transformersmentioning

confidence: 99%

When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism

Wang¹,

Zhao²,

Tang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Attention mechanism has been widely believed as the key to success of vision transformers (ViTs), since it provides a flexible and powerful way to model spatial relationships. However, is the attention mechanism truly an indispensable part of ViT? Can it be replaced by some other alternatives? To demystify the role of attention mechanism, we simplify it into an extremely simple case: ZERO FLOP and ZERO parameter. Concretely, we revisit the shift operation. It does not contain any parameter or arithmetic calculation. The only operation is to exchange a small portion of the channels between neighboring features. Based on this simple operation, we construct a new backbone network, namely ShiftViT, where the attention layers in ViT are substituted by shift operations. Surprisingly, ShiftViT works quite well in several mainstream tasks, e.g., classification, detection, and segmentation. The performance is on par with or even better than the strong baseline Swin Transformer. These results suggest that the attention mechanism might not be the vital factor that makes ViT successful. It can be even replaced by a zeroparameter operation. We should pay more attentions to the remaining parts of ViT in the future work. Code is available at github.com/microsoft/SPACH.

show abstract

Section: Related Work Attention and Vision Transformersmentioning

confidence: 99%

When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism

Wang¹,

Zhao²,

Tang³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Recently, the pioneering work ViT [22] successfully applies the pure transformer-based architecture to computer vision, revealing the potential of transformer in handling visual tasks. Lots of follow-up studies are proposed [4,5,9,12,18,21,23,24,[27][28][29]31,38,41,43,45,50,52,56,76,77,80,81,84]. Many of them analyze the ViT [15,17,26,32,44,55,69,73,75,82] and improve it via introducing locality to earlier layers [11,17,48,64,79,83,87].…”

Section: Related Workmentioning

confidence: 99%

ELSA: Enhanced Local Self-Attention for Vision Transformer

Zhou¹,

Wang²,

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

Self-attention is powerful in modeling long-range dependencies, but it is weak in local finer-level feature learning. The performance of local self-attention (LSA) is just on par with convolution and inferior to dynamic filters, which puzzles researchers on whether to use LSA or its counterparts, which one is better, and what makes LSA mediocre. To clarify these, we comprehensively investigate LSA and its counterparts from two sides: channel setting and spatial processing. We find that the devil lies in the generation and application of spatial attention, where relative position embeddings and the neighboring filter application are key factors. Based on these findings, we propose the enhanced local self-attention (ELSA) with Hadamard attention and the ghost head. Hadamard attention introduces the Hadamard product to efficiently generate attention in the neighboring case, while maintaining the high-order mapping. The ghost head combines attention maps with static matrices to increase channel capacity. Experiments demonstrate the effectiveness of ELSA. Without architecture / hyperparameter modification, drop-in replacing LSA with ELSA boosts Swin Transformer [48] by up to +1.4 on top-1 accuracy. ELSA also consistently benefits VOLO [83] from D1 to D5, where ELSA-VOLO-D5 achieves 87.2 on the ImageNet-1K without extra training images. In addition, we evaluate ELSA in downstream tasks. ELSA significantly improves the baseline by up to +1.9 box Ap / +1.3 mask Ap on the COCO, and by up to +1.9 mIoU on the ADE20K. Code is available at https://github.com/damo-cv/ELSA.

show abstract

“…Many attempts have been made to integrate the long-range modeling into CNNs, such as non-local networks [41,48], relation networks [21], etc. Vision Transformer (ViT) [12] first introduced a set of pure Transformer backbones for image classification, and its follow-ups modify the vision transformer soon to dominate many downstream tasks for computer vision, such as object detection [6,53], semantic segmentation [26], action recognition [3,14], 2D/3D human pose estimation [47,52], and 3D object detection [31]. It has shown great potential to be an alternative backbone for convolutional neural networks.…”

Section: Related Workmentioning

confidence: 99%

“…The vision transformer has achieved stunning success in computer vision since ViT [12]. It has shown impressive capability upon convolutional neural networks (CNNs) on prevalent visual domains, including image classification [9,39], object detection [5,53], semantic segmentation [26], action recognition [3,14] with both supervised and self-supervised [1,19] training configurations. Along with the development of ViT models, the deployment of vi- § Corresponding author.…”

Section: Introductionmentioning

confidence: 99%

Make A Long Image Short: Adaptive Token Length for Vision Transformers

Zhu¹,

Zhu²,

Du³

et al. 2021

Preprint

View full text Add to dashboard Cite

The vision transformer splits each image into a sequence of tokens with fixed length and processes the tokens in the same way as words in natural language processing. More tokens normally lead to better performance but considerably increased computational cost. Motivated by the proverb "A picture is worth a thousand words" we aim to accelerate the ViT model by making a long image short. To this end, we propose a novel approach to assign token length adaptively during inference. Specifically, we first train a ViT model, called Resizable-ViT (ReViT), that can process any given input with diverse token lengths. Then, we retrieve the "token-length label" from ReViT and use it to train a lightweight Token-Length Assigner (TLA). The token-length labels are the smallest number of tokens to split an image that the ReViT can make the correct prediction, and TLA is learned to allocate the optimal token length based on these labels. The TLA enables the ReViT to process the image with the minimum sufficient number of tokens during inference. Thus, the inference speed is boosted by reducing the token numbers in the ViT model. Our approach is general and compatible with modern vision transformer architectures and can significantly reduce computational expanse. We verified the effectiveness of our methods on multiple representative ViT models (DeiT [39], LV-ViT [22], and TimesFormer [3]) across two tasks (image classification and action recognition).

show abstract

Multiscale Vision Transformers

Cited by 69 publications

References 104 publications

When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism

When Shift Operation Meets Vision Transformer: An Extremely Simple Alternative to Attention Mechanism

ELSA: Enhanced Local Self-Attention for Vision Transformer

Make A Long Image Short: Adaptive Token Length for Vision Transformers

Contact Info

Product

Resources

About