GLiT: Neural Architecture Search for Global and Local Image Transformer

Chen, Boyu; Li, Peixia; Li, Chuming; Li, Baopu; Bai, Lei; Chen, Lin; Sun, Ming; Yan, Junjie; Ouyang, Wanli

doi:10.1109/iccv48922.2021.00008

Cited by 66 publications

(30 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The feature maps are firstly processed by a window-based local attention to aggregate information locally, and then passed through the deformable attention block to model the global relations between the local augmented tokens. This alternate design of attention blocks with local and global receptive fields helps the model learn strong representations, sharing a similar pattern in GLiT [5], TNT [15] and Point-…”

Section: Model Architecturesmentioning

confidence: 97%

Vision Transformer with Deformable Attention

Xia¹,

Pan²,

Song³

et al. 2022

Preprint

View full text Add to dashboard Cite

Transformers have recently shown superior performances on various vision tasks. The large, sometimes even global, receptive field endows Transformer models with higher representation power over their CNN counterparts. Nevertheless, simply enlarging receptive field also gives rise to several concerns. On the one hand, using dense attention e.g., in ViT, leads to excessive memory and computational cost, and features can be influenced by irrelevant parts which are beyond the region of interests. On the other hand, the sparse attention adopted in PVT or Swin Transformer is data agnostic and may limit the ability to model long range relations. To mitigate these issues, we propose a novel deformable self-attention module, where the positions of key and value pairs in self-attention are selected in a data-dependent way. This flexible scheme enables the self-attention module to focus on relevant regions and capture more informative features. On this basis, we present Deformable Attention Transformer, a general backbone model with deformable attention for both image classification and dense prediction tasks. Extensive experiments show that our models achieve consistently improved results on comprehensive benchmarks. Code is available at https://github.com/LeapLabTHU/DAT.

show abstract

Section: Model Architecturesmentioning

confidence: 97%

Vision Transformer with Deformable Attention

Xia¹,

Pan²,

Song³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Our method outperforms all of them at different target parameters and FLOPs. GLiT-S [6] improves accuracy (80.5%) over baseline DeiT-S but with an additional parameter increase and minimal FLOPs reduction. Our ViT- .…”

Section: Comparison To State-of-the-art Approachesmentioning

confidence: 96%

“…Due to the adaptation of evolutionary algorithm, a large search space cannot be explored. GLiT [6] introduces locality module to model local features along with the global features. But their method uses CNNs along with attention and performs an evolutionary search over global and local modules.…”

Section: Related Workmentioning

confidence: 99%

“…Recent works have followed two methodologies for defining the search space for MHSA module -1) Searching for the number of heads in every distinct MHSA module [8,25] and/or 2) Searching for a common feature dimension size from a pre-defined discrete sample space for all attention heads in any particular MHSA module [6,7,42]. These methods have shown some solid results but they are not completely flexible.…”

Section: Flexible Mhsa and Mlp Modulesmentioning

confidence: 99%

See 1 more Smart Citation

Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space

Chavan¹,

Shen²,

Liu³

et al. 2022

Preprint

View full text Add to dashboard Cite

This paper explores the feasibility of finding an optimal sub-model from a vision transformer and introduces a pure vision transformer slimming (ViT-Slim) framework that can search such a sub-structure from the original model endto-end across multiple dimensions, including the input tokens, MHSA and MLP modules with state-of-the-art performance. Our method is based on a learnable and unified 1 sparsity constraint with pre-defined factors to reflect the global importance in the continuous searching space of different dimensions. The searching process is highly efficient through a single-shot training scheme. For instance, on DeiT-S, ViT-Slim only takes ∼43 GPU hours for searching process, and the searched structure is flexible with diverse dimensionalities in different modules. Then, a budget threshold is employed according to the requirements of accuracy-FLOPs trade-off on running devices, and a retraining process is performed to obtain the final models. The extensive experiments show that our ViT-Slim can compress up to 40% of parameters and 40% FLOPs on various vision transformers while increasing the accuracy by ∼0.6% on ImageNet. We also demonstrate the advantage of our searched models on several downstream datasets. Our source code will be publicly available later.

show abstract

“…As a result, we can eliminate some costly MSA layers to significantly improve the efficiency for ViT models while adding proper locality without heuristics. Recently, several one-shot NAS methods propose to include candidate MSA and convolutional operations separately to the search space [9,33,60], where each operation is formulated as a separate trainable path (see Figure 1 (a)). However, these multi-path methods can give rise to the expensive search cost and challenging optimization since all the candidate operations for each layer need to be maintained and updated independently during the search.…”

mentioning

confidence: 99%

Pruning Self-attentions into Convolutional Layers in Single Path

He¹,

Liu²,

Pan³

et al. 2021

Preprint

View full text Add to dashboard Cite

Vision Transformers (ViTs) have achieved impressive performance over various computer vision tasks. However, modeling global correlations with multi-head self-attention (MSA) layers leads to two widely recognized issues: the massive computational resource consumption and the lack of intrinsic inductive bias for modeling local visual patterns. One unified solution is to search whether to replace some MSA layers with convolution-like inductive biases that are computationally efficient via neural architecture search (NAS) based pruning methods. However, maintaining MSA and different candidate convolutional operations as separate trainable paths gives rise to expensive search cost and challenging optimization. Instead, we propose a novel weight-sharing scheme between MSA and convolutional operations and cast the search problem as finding which subset of parameters to use in each MSA layer. The weightsharing scheme further allows us to devise an automatic Single-Path Vision Transformer pruning method (SPViT) to quickly prune the pre-trained ViTs into accurate and compact hybrid models with significantly reduced search cost, given target efficiency constraints. We conduct extensive experiments on two representative ViT models showing our method achieves a favorable accuracy-efficiency trade-off. Code is available at https://github.com/zhuanggroup/SPViT.

show abstract

GLiT: Neural Architecture Search for Global and Local Image Transformer

Cited by 66 publications

References 18 publications

Vision Transformer with Deformable Attention

Vision Transformer with Deformable Attention

Vision Transformer Slimming: Multi-Dimension Searching in Continuous Optimization Space

Pruning Self-attentions into Convolutional Layers in Single Path

Contact Info

Product

Resources

About