Going deeper with Image Transformers

Touvron, Hugo; Cord, Matthieu; Sablayrolles, Alexandre; Synnaeve, Gabriel

doi:10.48550/arxiv.2103.17239

Cited by 38 publications

(49 citation statements)

References 52 publications

(107 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Under Isotrpic designs, DeiT-B [28] suffers from high memory requirement although being only a 12-layer network. Our performance match with the CaiT-S24 [29] answers our research question by successfully encapsulating the information into the lower number of tokens (Super tokens) without degrading performance.…”

Section: Results On Imagenet-1ksupporting

confidence: 57%

“…Both the input and output of the transformer have the same dimensions x f , y ∈ R n×D . The earlier works ViT [11], Deit [28] and CaiT [29] followed this design. The Deit [16] introduced distillation from the CNN teacher network while the CaiT [29] proposed a layer scaling operation to aid with increasing depth of the transformers.…”

Section: Related Workmentioning

confidence: 99%

“…However, we demonstrate in the ablation study, that only the first global interaction is needed after learning the initial few layers locally. We adopted the per-channel LayerScale (λ i ) from [29] to dynamically scale each information channel. The mathematical formulation of the STT-data encoder at the layer l is as follows:…”

Section: Global Interaction With Super Token Mixermentioning

confidence: 99%

“…We follow CaiT [29] learning schedule to train our models. We use batch size 512 with base learning rate 0.0005 and five warm-up epochs.…”

Section: Implementation Detailsmentioning

confidence: 99%

“…In CV, the number of data tokens is fairly high, due to the 2D spatial nature of the input. Several techniques have been proposed to address the computational complexity of transformers [3,5,12,13,16,18,21,29,32,35]. Window-based self-attention (WMSA) has become one of the most popular mechanisms to reduce computational complexity of the self-attention operation [5,21,32].…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Global Interaction Modelling in Vision Transformer via Super Tokens

Farooq¹,

Awais²,

Ahmed³

et al. 2021

Preprint

View full text Add to dashboard Cite

With the popularity of Transformer architectures in computer vision, the research focus has shifted towards developing computationally efficient designs. Window-based local attention is one of the major techniques being adopted in recent works. These methods begin with very small patch size and small embedding dimensions and then perform strided convolution (patch merging) in order to reduce the feature map size and increase embedding dimensions, hence, forming a pyramidal Convolutional Neural Network (CNN) like design. In this work, we investigate local and global information modelling in transformers by presenting a novel isotropic architecture that adopts local windows and special tokens, called Super tokens, for self-attention. Specifically, a single Super token is assigned to each image window which captures the rich local details for that window. These tokens are then employed for cross-window communication and global representation learning. Hence, most of the learning is independent of the image patches (N ) in the higher layers, and the class embedding is learned solely based on the Super tokens (N M 2 ) where M 2 is the window size. In standard image classification on Imagenet-1K, the proposed Super tokens based transformer (STT-S25) achieves 83.5% accuracy which is equivalent to Swin transformer (Swin-B) with circa half the number of parameters (49M) and double the inference time throughput. The proposed Super token transformer offers a lightweight and promising backbone for visual recognition tasks.

show abstract

Section: Results On Imagenet-1ksupporting

confidence: 57%

Section: Related Workmentioning

confidence: 99%

Section: Global Interaction With Super Token Mixermentioning

confidence: 99%

“…We follow CaiT [29] learning schedule to train our models. We use batch size 512 with base learning rate 0.0005 and five warm-up epochs.…”

Section: Implementation Detailsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Global Interaction Modelling in Vision Transformer via Super Tokens

Farooq¹,

Awais²,

Ahmed³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

Airborne Sensor and Perception Management

Ruß

Stütz

2023

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Improving new physics searches with diffusion models for event observables and jet constituents

Sengupta,

Leigh,

Raine

et al. 2024

J. High Energ. Phys.

View full text Add to dashboard Cite

We introduce a new technique called Drapes to enhance the sensitivity in searches for new physics at the LHC. By training diffusion models on side-band data, we show how background templates for the signal region can be generated either directly from noise, or by partially applying the diffusion process to existing data. In the partial diffusion case, data can be drawn from side-band regions, with the inverse diffusion performed for new target conditional values, or from the signal region, preserving the distribution over the conditional property that defines the signal region. We apply this technique to the hunt for resonances using the LHCO di-jet dataset, and achieve state-of-the-art performance for background template generation using high level input features. We also show how Drapes can be applied to low level inputs with jet constituents, reducing the model dependence on the choice of input observables. Using jet constituents we can further improve sensitivity to the signal process, but observe a loss in performance where the signal significance before applying any selection is below 4σ.

show abstract

Going deeper with Image Transformers

Cited by 38 publications

References 52 publications

Global Interaction Modelling in Vision Transformer via Super Tokens

Global Interaction Modelling in Vision Transformer via Super Tokens

Airborne Sensor and Perception Management

Improving new physics searches with diffusion models for event observables and jet constituents

Contact Info

Product

Resources

About