Logarithmic Unbiased Quantization: Simple 4-bit Training in Deep Learning

Abstract

Quantization of the weights and activations is one of the main methods to reduce the computational footprint of Deep Neural Networks (DNNs) training. Current methods enable 4-bit quantization of the forward phase. However, this constitutes only a third of the training process. Reducing the computational footprint of the entire training process requires the quantization of the neural gradients, i.e., the loss gradients with respect to the outputs of intermediate neural layers. In this work, we examine the importance of having unbiased quantization in quantized neural network training, where to maintain it, and how. Based on this, we suggest a logarithmic unbiased quantization (LUQ) method to quantize both the forward and backward phase to 4-bit, achieving state-of-the-art results in 4-bit training without overhead. For example, in ResNet50 on ImageNet, we achieved a degradation of 1.18%. We further improve this to degradation of only 0.64% after a single epoch of high precision fine-tuning combined with a variance reduction method-both add overhead comparable to previously suggested methods. Finally, we suggest a method that uses the low precision format to avoid multiplications during two-thirds of the training process, thus reducing by 5x the area used by the multiplier.

Low-precision logarithmic arithmetic for neural network accelerators

“…Several other authors have proposed full training strategies [22,20,34,35], but for bitcounts over 8. [36] proposes an approach using 4 bits, but fixes the representation, while our neuron is parametric, which will offer more flexibility. For such a study, the proposed approach also has the potential to use a different base b and an arbitrary activation function, provided that the training ensures that weights and activations remain below 1.…”

Ultra-low Precision Multiplication-free Training for Deep Neural Networks

“…For example, the energy consumption of an INT32 multiplication is approximately 22x higher than that of an INT32 addition. Thus, there are multiplication-less methods that directly replace the multiplication with energy-efficient operations such as addition and bitwise shift [7,8,13,15,24,37,38]. Most of these works, such as INQ [38], ShiftCNN [15], and LogNN [24] also start with the FP32 pre-trained models rather than training from scratch so they cannot reduce the energy consumption of training.…”

“…Most of these works, such as INQ [38], ShiftCNN [15], and LogNN [24] also start with the FP32 pre-trained models rather than training from scratch so they cannot reduce the energy consumption of training. Among the methods that can train from scratch [7,8,13], AdderNet [7] replaces all of the FP32 multiplications in the linear layer with FP32 additions whose energy consumption is still higher than the fixed point operations. The other works [8,13] apply low-precision Powerof-Two (PoT) numbers, whose value is zero or power of 2, to replace a part of the multiplications in training with bitwise shifts and sign flip operations.…”

“…Among the methods that can train from scratch [7,8,13], AdderNet [7] replaces all of the FP32 multiplications in the linear layer with FP32 additions whose energy consumption is still higher than the fixed point operations. The other works [8,13] apply low-precision Powerof-Two (PoT) numbers, whose value is zero or power of 2, to replace a part of the multiplications in training with bitwise shifts and sign flip operations. However, they cannot replace all of the multiplications during forward or backward propagation.…”

“…For example, DeepShift [13] only converts W to 5-bit PoT numbers because only the value range of W can be limited to the representation range of their 5-bit PoT numbers. Similarly, LUQ [8] only converts G to PoT numbers because their method cannot approximate distributions of W and A well. These methods cannot use one data format to represent each of W , A, and G whose data ranges and distributions vary widely from each other, so they keep one-third of the multiplications in training.…”