Patch Similarity Aware Data-Free Quantization for Vision Transformers

Li, Zhikai; Ma, Lan; Chen, Mengjuan; Xiao, Junrui; Gu, Qingyi

doi:10.48550/arxiv.2203.02250

Cited by 2 publications

(4 citation statements)

References 31 publications

(56 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Furthermore, ViTs have also been applied to more complexed vision applications, such as object detection [2,27] and semantic segmentation [3]. Despite the promising performance, ViTs' complicated architectures with large memory footprints and computational overheads is intolerable in real-world applications [15,19], especially in time/resource-constrained scenarios. Thus, the compression approaches for ViTs are necessary for practical deployments.…”

Section: Related Work 21 Vision Transformersmentioning

confidence: 99%

“…Ranking loss [19] is presented to maintain the correct relative order of the quantized attention map. PSAQ-ViT [15] pushes the quantization of ViTs to data-free scenarios based on patch similarity. To realize the full quantization of ViTs, FQ-ViT [17] introduces quantization strategies for LayerNorm and Softmax.…”

Section: Model Quantizationmentioning

confidence: 99%

“…Vision Transformers (ViTs) have recently achieved great success on a variety of computer vision tasks [2,6,8]. Nevertheless, as compared to convolutional neural networks (CNNs), ViTs suffer from higher memory footprints, computational overheads, and power consumption, hindering their deployment and real-time inference on resource-constrained edge devices [15,17,19]. Thus, compression approaches for ViTs are being widely researched.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference

Li¹,

Gu²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

Vision Transformers (ViTs) have achieved state-of-the-art performance on various computer vision applications. These models, however, have considerable storage and computational overheads, making their deployment and efficient inference on edge devices challenging. Quantization is a promising approach to reducing model complexity; unfortunately, existing efforts to quantize ViTs are simulated quantization (aka fake quantization), which remains floating-point arithmetic during inference and thus contributes little to model acceleration. In this paper, we propose I-ViT, an integeronly quantization scheme for ViTs, to enable ViTs to perform the entire computational graph of inference with integer operations and bit-shifting and no floating-point operations. In I-ViT, linear operations (𝑒.𝑔., MatMul and Dense) follow the integer-only pipeline with dyadic arithmetic, and non-linear operations (𝑒.𝑔., Softmax, GELU, and LayerNorm) are approximated by the proposed lightweight integer-only arithmetic methods. In particular, I-ViT applies the proposed Shiftmax and ShiftGELU, which are designed to use integer bit-shifting to approximate the corresponding floating-point operations. We evaluate I-ViT on various benchmark models and the results show that integer-only INT8 quantization achieves comparable (or even higher) accuracy to the full-precision (FP) baseline. Furthermore, we utilize TVM for practical hardware deployment on the GPU's integer arithmetic units, achieving 3.72~4.11× inference speedup compared to the FP model.

show abstract

Section: Related Work 21 Vision Transformersmentioning

confidence: 99%

Section: Model Quantizationmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference

Li¹,

Gu²

2022

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…This is best illustrated on ViT where PowerQuant W4/A8 out performs both DFQ and SQuant even when they are allowed 8 bits for the weights (W8/A8) by a whopping 4.91 points. The proposed PowerQuant even outperforms methods dedicated to transformer quantization such as PSAQLi et al (2022) on every image transformer tested.…”

mentioning

confidence: 95%

PowerQuant: Automorphism Search for Non-Uniform Quantization

Yvinec¹,

Dapogny²,

Cord³

et al. 2023

Preprint

View full text Add to dashboard Cite

Deep neural networks (DNNs) are nowadays ubiquitous in many domains such as computer vision. However, due to their high latency, the deployment of DNNs hinges on the development of compression techniques such as quantization which consists in lowering the number of bits used to encode the weights and activations. Growing concerns for privacy and security have motivated the development of data-free techniques, at the expanse of accuracy. In this paper, we identity the uniformity of the quantization operator as a limitation of existing approaches, and propose a data-free non-uniform method. More specifically, we argue that to be readily usable without dedicated hardware and implementation, non-uniform quantization shall not change the nature of the mathematical operations performed by the DNN. This leads to search among the continuous automorphisms of (R * + , ×), which boils down to the power functions defined by their exponent. To find this parameter, we propose to optimize the reconstruction error of each layer: in particular, we show that this procedure is locally convex and admits a unique solution. At inference time, we show that our approach, dubbed PowerQuant, only require simple modifications in the quantized DNN activation functions. As such, with only negligible overhead, it significantly outperforms existing methods in a variety of configurations.

show abstract

Patch Similarity Aware Data-Free Quantization for Vision Transformers

Cited by 2 publications

References 31 publications

I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference

I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference

PowerQuant: Automorphism Search for Non-Uniform Quantization

Contact Info

Product

Resources

About