How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers

Abstract

Vision Transformers (ViT) have been shown to attain highly competitive performance for a wide range of vision applications, such as image classification, object detection and semantic image segmentation. In comparison to convolutional neural networks, the Vision Transformer's weaker inductive bias is generally found to cause an increased reliance on model regularization or data augmentation ("AugReg" for short) when training on smaller training datasets. We conduct a systematic empirical study in order to better understand the interplay between the amount of training data, AugReg, model size and compute budget. 1 As one result of this study we find that the combination of increased compute and AugReg can yield models with the same performance as models trained on an order of magnitude more training data: we train ViT models of various sizes on the public ImageNet-21k dataset which either match or outperform their counterparts trained on the larger, but not publicly available JFT-300M dataset.Preprint. * Equal contribution.

Multiview Transformers for Video Recognition

“…For the backbone of each view, we consider five ViT variants, "Tiny", "Small", "Base", "Large", and "Huge". Their settings strictly follow the ones defined in BERT [17] and ViT [18,60], i.e. number of transformer layers, number of attention heads, hidden dimensions.…”

“…All model variants use the same global encoder which follows the "Base" architecture, except that the number of heads is set to 8 instead of 12. The reason is that the hidden dimension of the tokens should be divisible by the number of heads for multi-head attention, and the number of hidden dimensions across all standard transformer architectures (from "Tiny" to "Huge" [18,59]) is divisible by 8.…”

“…For the backbone of each view, we consider five ViT variants, "Tiny", "Small","Base", "Large", and "Huge". Their settings strictly follow the ones defined in BERT [17] and ViT [18,60]. For the global encoder, all model variants of MTV use the same global encoder which follows the "Base" architecture, except that the number of heads is set to 8 instead of 12.…”

How to augment your ViTs? Consistency loss and StyleAug, a random style transfer augmentation

“…Although ViTs have attained competitive performance on vision tasks, they are known to be more difficult to train than CNNs (Steiner et al, 2021;. In ViTs, only multi-layer perceptron (MLP) layers operate locally and are translationally equivariant, while the self-attention layers (Vaswani et al, 2017) operate globally (Dosovitskiy et al, 2021).…”

“…In ViTs, only multi-layer perceptron (MLP) layers operate locally and are translationally equivariant, while the self-attention layers (Vaswani et al, 2017) operate globally (Dosovitskiy et al, 2021). As such, ViTs are thought to have weaker inductive biases than CNNs, thus requiring more data, augmentations, and/or regularization than training similarly-sized CNNs Steiner et al, 2021;. However, the strategies for data augmentation for training ViTs have largely been adapted from the techniques used for CNNs.…”