FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformer

Abstract

Network quantization significantly reduces model inference complexity and has been widely used in real-world deployments. However, most existing quantization methods have been developed and tested mainly on Convolutional Neural Networks (CNN), and suffer severe degradation when applied to Transformer-based architectures. In this work, we present a systematic method to reduce the performance degradation and inference complexity of Quantized Transformers. In particular, we propose Powers-of-Two Scale (PTS) to deal with the serious inter-channel variation of LayerNorm inputs in a hardware-friendly way. In addition, we propose Log-Int-Softmax (LIS) that can sustain the extreme non-uniform distribution of the attention maps while simplifying inference by using 4-bit quantization and the BitShift operator. Comprehensive experiments on various Transformer-based architectures and benchmarks show that our methods outperform previous works in performance while using even lower bit-width in attention maps. For instance, we reach 85.17% Top-1 accuracy with ViT-L on ImageNet and 51.4 mAP with Cascade Mask R-CNN (Swin-S) on COCO. To our knowledge, we are the first to achieve comparable accuracy degradation (∼1%) on fully quantized Vision Transformers. Code is available at https://github.com/linyang-zhh/FQ-ViT.

Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey

“…FQ-ViT [43] adopts log2 quantization (see eq.9) and assigns more bins to the frequently occurring small values found in post-softmax activation (attention maps), in contrast to the 4-bit uniform quantization, which allocates only one bin for these values.…”

“…2) Post-LayerNorm Activation: FQ-ViT [43] proposes the Power-of-Two Factor (PTF) for Pre-LayerNorm quantization. The core idea of PTF is to assign different factors to different channels instead of different quantization parameters.…”

“…4) Outlier-Aware Training: PackQViT [69] introduces an outlier-aware training approach, differing from the method in [43] which calculates the power-of-two factor during calibration. In each training iteration of PackQViT, the process begins by searching for the channel index and the power-oftwo coefficient of each outlier using l 2 minimization.…”

“…FQ-ViT [43] introduces "Log-Int-Softmax," an integeronly variant of the softmax function that approximates the exponential component using a second-order polynomial [74]. This method is coupled with Log2 quantization for efficient computation.…”

“…This method employs log2 quantization post-exponentiation and an approximate logarithmic division, eschewing traditional high-precision operations. Additionally, SOLE employs dynamic compression for Layer Normalization statistics via Power-of-Two Factor [43], and innovatively designs a two-stage LayerNorm Unit. This design facilitates pipelined computations and flexible batch processing, with robust error tolerance and reduced buffering and multiplication.…”