Escaping the Big Data Paradigm with Compact Transformers

Hassani, Ali; Walton, Steven A.; Shah, Nikhil; Abuduweili, Abulikemu; Li, Jiachen

doi:10.48550/arxiv.2104.05704

Cited by 98 publications

(153 citation statements)

References 40 publications

Supporting

Mentioning

152

Contrasting

Order By: Relevance

“…Shuffle Swin Transformer [15] proposes shuffle multi-headed attention to augment spatial connection between windows. CCT [7] proposes a convolutional tokenizer and compact vision transformers, leading to better performance on smaller datasets training from scratch, with fewer parameters compared with ViT. TransCNN [23] also proposes a co-design of convolutions and multi-headed attention to learn hierarchical representations.…”

Section: Related Workmentioning

confidence: 99%

ConvMLP: Hierarchical Convolutional MLPs for Vision

Li¹,

Hassani²,

Walton³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

MLP-based architectures, which consist of a sequence of consecutive multi-layer perceptron blocks, have recently been found to reach comparable results to convolutional and transformer-based methods. However, most adopt spatial MLPs which take fixed dimension inputs, therefore making it difficult to apply them to downstream tasks, such as object detection and semantic segmentation. Moreover, single-stage designs further limit performance in other computer vision tasks and fully connected layers bear heavy computation. To tackle these problems, we propose Con-vMLP: a hierarchical Convolutional MLP for visual recognition, which is a light-weight, stage-wise, co-design of convolution layers, and MLPs. In particular, ConvMLP-S achieves 76.8% top-1 accuracy on ImageNet-1k with 9M parameters and 2.4 GMACs (15% and 19% of MLP-Mixer-B/16, respectively). Experiments on object detection and semantic segmentation further show that visual representation learned by ConvMLP can be seamlessly transferred and achieve competitive results with fewer parameters. Our code and pre-trained models are publicly available at https://github.

show abstract

Section: Related Workmentioning

confidence: 99%

ConvMLP: Hierarchical Convolutional MLPs for Vision

Li¹,

Hassani²,

Walton³

2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…We evaluate the average performance of NFM with different model architectures on CIFAR-10 [36], CIFAR-100 [36], and ImageNet [13]. We use a pre-activated residual network (ResNet) with depth 18 [29] and a compact vision transformer (ViT-lite) with 7 attention layers and 4 heads [28] on small scale tasks. For more challenging and higher dimensional tasks, we consider the performance of wide ResNet-18 [77] and ResNet-50 architectures, respectively.…”

Section: Resultsmentioning

confidence: 99%

Noisy Feature Mixup

Lim¹,

Erichson²,

Utrera³

et al. 2021

Preprint

View full text Add to dashboard Cite

We introduce Noisy Feature Mixup (NFM), an inexpensive yet effective method for data augmentation that combines the best of interpolation based training and noise injection schemes. Rather than training with convex combinations of pairs of examples and their labels, we use noise-perturbed convex combinations of pairs of data points in both input and feature space. This method includes mixup and manifold mixup as special cases, but it has additional advantages, including better smoothing of decision boundaries and enabling improved model robustness. We provide theory to understand this as well as the implicit regularization effects of NFM. Our theory is supported by empirical results, demonstrating the advantage of NFM, as compared to mixup and manifold mixup. We show that residual networks and vision transformers trained with NFM have favorable trade-offs between predictive accuracy on clean data and robustness with respect to various types of data perturbation across a range of computer vision benchmark datasets.

show abstract

“…LV-ViT [24] is equipped with a token labeling method. CCT [16] is constructed with convolutions and a sequence pooling strategy. These two works both significantly reduced the number of parameters.…”

Section: Vision Transformermentioning

confidence: 99%

FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer

Yang¹,

Zhang²,

Sun³

et al. 2021

Preprint

View full text Add to dashboard Cite

Network quantization significantly reduces model inference complexity and has been widely used in real-world deployments. However, most existing quantization methods have been developed and tested mainly on Convolutional Neural Networks (CNN), and suffer severe degradation when applied to Transformer-based architectures. In this work, we present a systematic method to reduce the performance degradation and inference complexity of Quantized Transformers. In particular, we propose Powers-of-Two Scale (PTS) to deal with the serious inter-channel variation of LayerNorm inputs in a hardware-friendly way. In addition, we propose Log-Int-Softmax (LIS) that can sustain the extreme non-uniform distribution of the attention maps while simplifying inference by using 4-bit quantization and the BitShift operator. Comprehensive experiments on various Transformer-based architectures and benchmarks show that our methods outperform previous works in performance while using even lower bit-width in attention maps. For instance, we reach 85.17% Top-1 accuracy with ViT-L on ImageNet and 51.4 mAP with Cascade Mask R-CNN (Swin-S) on COCO. To our knowledge, we are the first to achieve comparable accuracy degradation (∼1%) on fully quantized Vision Transformers. Code is available at https://github.com/linyang-zhh/FQ-ViT.

show abstract

Escaping the Big Data Paradigm with Compact Transformers

Cited by 98 publications

References 40 publications

ConvMLP: Hierarchical Convolutional MLPs for Vision

ConvMLP: Hierarchical Convolutional MLPs for Vision

Noisy Feature Mixup

FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer

Contact Info

Product

Resources

About