Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

Li, Yuan; Chen, Yunpeng; Wang, Tao; Yu, Weihao; Shi, Yong; Jiang, Zihang; Tay, Francis Eng Hock; Feng, Jiashi; Yan, Shuicheng

doi:10.1109/iccv48922.2021.00060

Cited by 1,529 publications

(479 citation statements)

References 21 publications

Supporting

Mentioning

386

Contrasting

Order By: Relevance

“…non-overlapped). However, as pointed out by prior works like T2T [51], CrossViT [5], and PiT [19], using a stronger but still simple subnetwork for the tokenization could further improve the performance, especially for the smaller models. Thus, we adopt one convolutional layer with overlapped kernel and identical stride when generating local tokens, e.g.…”

Section: Rsa Regional Tokensmentioning

confidence: 99%

“…Motivated by this, two main line of research works have been developed to improve ViT. One is to enhance different components of the vision transformer [51,16,41,23] while still using isotropic structure (i.e., fixed token numbers and channel dimension) like ViT, e.g. while T2T-ViT [51] introduce an Tokens-to-Token (T2T) transformation to encode the important local structure for each token instead of the naive tokenization, CrossViT propose a dual-path architecture, each with different scales, to learn multi-scale feature.…”

Section: Related Workmentioning

confidence: 99%

“…One is to enhance different components of the vision transformer [51,16,41,23] while still using isotropic structure (i.e., fixed token numbers and channel dimension) like ViT, e.g. while T2T-ViT [51] introduce an Tokens-to-Token (T2T) transformation to encode the important local structure for each token instead of the naive tokenization, CrossViT propose a dual-path architecture, each with different scales, to learn multi-scale feature. CaiT [41] propose a layer-scalar to assure training a deeper network for better performance and LV-ViT [23] modified how the model is trained when CutMix [52] augmentation is applied on ViT.…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

RegionViT: Regional-to-Local Attention for Vision Transformers

Chen¹,

Panda²,

Fan³

2021

Preprint

View full text Add to dashboard Cite

Vision transformer (ViT) has recently showed its strong capability in achieving comparable results to convolutional neural networks (CNNs) on image classification. However, vanilla ViT simply inherits the same architecture from the natural language processing directly, which is often not optimized for vision applications. Motivated by this, in this paper, we propose a new architecture that adopts the pyramid structure and employ a novel regional-to-local attention rather than global self-attention in vision transformers. More specifically, our model first generates regional tokens and local tokens from an image with different patch sizes, where each regional token is associated with a set of local tokens based on the spatial location. The regional-to-local attention includes two steps: first, the regional selfattention extract global information among all regional tokens and then the local self-attention exchanges the information among one regional token and the associated local tokens via self-attention. Therefore, even though local self-attention confines the scope in a local region but it can still receive global information. Extensive experiments on three vision tasks, including image classification, object detection and action recognition, show that our approach outperforms or is on par with state-of-the-art ViT variants including many concurrent works. Our source codes and models will be publicly available.Preprint. Under review.

show abstract

Section: Rsa Regional Tokensmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

RegionViT: Regional-to-Local Attention for Vision Transformers

Chen¹,

Panda²,

Fan³

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Furthermore, knowledge distillation Hinton et al (2015) ; Wei et al (2020) from the model based on CNN is shown to be effective in improving the performance of the vision transformer Touvron et al (2020) . Instead of simply regarding image patches as tokens, Yuan et al proposed a tokens-to-token (T2T) method to better tokenize patches with the consideration of image structure Yuan et al (2021) . The T2T method achieves better accuracy using fewer parameters compared with the vanilla vision transformer Dosovitskiy et al (2020) .…”

Section: Related Workmentioning

confidence: 99%

Classification of Diabetic Foot Ulcers Using Class Knowledge Banks

Han

Zhou

et al. 2022

Front. Bioeng. Biotechnol.

View full text Add to dashboard Cite

Diabetic foot ulcers (DFUs) are one of the most common complications of diabetes. Identifying the presence of infection and ischemia in DFU is important for ulcer examination and treatment planning. Recently, the computerized classification of infection and ischaemia of DFU based on deep learning methods has shown promising performance. Most state-of-the-art DFU image classification methods employ deep neural networks, especially convolutional neural networks, to extract discriminative features, and predict class probabilities from the extracted features by fully connected neural networks. In the testing, the prediction depends on an individual input image and trained parameters, where knowledge in the training data is not explicitly utilized. To better utilize the knowledge in the training data, we propose class knowledge banks (CKBs) consisting of trainable units that can effectively extract and represent class knowledge. Each unit in a CKB is used to compute similarity with a representation extracted from an input image. The averaged similarity between units in the CKB and the representation can be regarded as the logit of the considered input. In this way, the prediction depends not only on input images and trained parameters in networks but the class knowledge extracted from the training data and stored in the CKBs. Experimental results show that the proposed method can effectively improve the performance of DFU infection and ischaemia classifications.

show abstract

“…Recently, the Transformer neural network architecture was proposed [19] for sequential data, with great success in natural language processing [14,4], and more recently, vision [5,3,6,21,18]. The attention mechanism is at the core of Transformer, which is readily capable of learning long-range dependencies between any two positions in the input data in the form of an attention map.…”

Section: Learning Architectures For Visual Tasksmentioning

confidence: 99%

Can Attention Enable MLPs To Catch Up With CNNs?

Guo

Liu

et al. 2021

Preprint

View full text Add to dashboard Cite

In the first week of May, 2021, researchers from four different institutions: Google, Tsinghua University, Oxford University and Facebook, shared their latest work [16,7,12,17] on arXiv.org almost at the same time, each proposing new learning architectures, consisting mainly of linear layers, claiming them to be comparable, or even superior to convolutional-based models. This sparked immediate discussion and debate in both academic and industrial communities as to whether MLPs are sufficient, many thinking that learning architectures are returning to MLPs. Is this true?In this perspective, we give a brief history of learning architectures, including multilayer perceptrons (MLPs), convolutional neural networks (CNNs) and transformers. We then examine what the four newly proposed architectures have in common. Finally, we give our views on challenges and directions for new learning architectures, hoping to inspire future research.

show abstract

Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet

Cited by 1,529 publications

References 21 publications

RegionViT: Regional-to-Local Attention for Vision Transformers

RegionViT: Regional-to-Local Attention for Vision Transformers

Classification of Diabetic Foot Ulcers Using Class Knowledge Banks

Can Attention Enable MLPs To Catch Up With CNNs?

Contact Info

Product

Resources

About