Post-training 4-bit quantization of convolution networks for rapid-deployment

Abstract

Convolutional neural networks require significant memory bandwidth and storage for intermediate computations, apart from substantial computing resources. Neural network quantization has significant benefits in reducing the amount of intermediate results, but it often requires the full datasets and time-consuming fine tuning to recover the accuracy lost after quantization. This paper introduces the first practical 4-bit post training quantization approach: it does not involve training the quantized model (fine-tuning), nor it requires the availability of the full dataset. We target the quantization of both activations and weights and suggest three complementary methods for minimizing quantization error at the tensor level, two of whom obtain a closed-form analytical solution. Combining these methods, our approach achieves accuracy that is just a few percents less the state-of-the-art baseline across a wide range of convolutional models. The source code to replicate all experiments is available on GitHub: https://github.com/submission2019/cnn-quantization.

Post-training Quantization for Neural Networks with Provable Guarantees

“…Hence, much attention has recently been dedicated to post-training quantization schemes, which directly quantize pretrained DNNs, with real-valued weights, without retraining. These quantization methods either rely on a small amount of data [1,3,35,24,14,31,19,22] or can be implemented without accessing training data, i.e. data-free compression [23,2,33,20].…”

“…Let S ⊆ R n be a Borel set. Unif(S) denotes the uniform distribution over S. A L-layer multi-layer perceptron, Φ, acts on a vector x ∈ R N 0 via (1) Φ(x) := ϕ (L) • A (L) • • • • • ϕ (1) • A (1) (x)…”

“…∈ Z} where δ > 0 denotes the quantization step size. For example, A 2 1 = {±1} is a 1-bit alphabet while ‹ A 1 1 = {0, ±1} is a ternary alphabet. Moreover, the following augmented midrise alphabets can also be used in practice: (4) Âδ K := A δ K ∪ {0}.…”

“…Nevertheless, these works use different strategies for determining the quantization bins. For example, Banner et al [1] (see also [35]) choose the thresholds to minimize a MSE metric. Their numerical results also show that for convolutional networks using different quantization thresholds "per-channel" and bias correction can improve the accuracy of quantized models.…”

Bit-serial Weight Pools: Compression and Arbitrary Precision Execution of Neural Networks on Resource Constrained Processors

“…Since weight sharing does not modify the structure nor the precision of the network, it can be combined with other compression techniques like pruning and quantization to further improve compression ratio and runtime. Furthermore, recent works (Choi et al, 2018;Banner et al, 2018) have shown that sub-byte quantization of weights and/or activations can achieve inference accuracies comparable to full-precision networks.…”