Neural gradients are near-lognormal: improved quantized and sparse training

Abstract

Neural gradient compression remains a main bottleneck in improving training efficiency, as most existing neural network compression methods (e.g., pruning or quantization) focus on weights, activations, and weight gradients. However, these methods are not suitable for compressing neural gradients, which have a very different distribution. Specifically, we find that the neural gradients follow a lognormal distribution. Taking this into account, we suggest two methods to reduce the computational and memory burdens of neural gradients. The first one is stochastic gradient pruning, which can accurately set the sparsity level -up to 85% gradient sparsity without hurting validation accuracy (ResNet18 on ImageNet). The second method determines the floating-point format for low numerical precision gradients (e.g., FP8). Our results shed light on previous findings related to local scaling, the optimal bit-allocation for the mantissa and exponent, and challenging workloads for which low-precision floating-point arithmetic has reported to fail. Reference implementation accompanies the paper. * Equal contribution.Preprint. Under review.

The Future is Log-Gaussian: ResNets and Their Infinite-Depth-and-Width Limit at Initialization

“…Our result shows that ResNets have log-Gaussian behaviour on initialization, and like fully connected networks [22], the behaviour is determined by the depth-to-width aspect ratio. This corroborates recent empirical observations about deep ResNets [25]. Since real world networks are finite, the question of how well this approximates finite behaviour is of paramount importance.…”

“…The input-output derivative ∂ xi z out has the same type of behaviour as z out itself; a simple proof is given in Appendix B. It is expected that the gradient with respect to the weights ∂ W ij z out will also have the same qualitative behaviour [25] although more investigation is needed to understand this theoretically. Exponentially large variance for gradients is a manifestation of the vanishing-and-exploding gradient problem [28].…”

“…In addition to these theoretical corrections to the Gaussian limit, practitioners have begun to notice that even basic properties of standard neural networks do not line up with those predicted by the infinite-width limit. Perhaps the most basic is that the gradients of the network early in training are not Gaussian, but instead are approximately log-Gaussian [25]. In fact, one should see log-Gaussian behaviour agrees with the theoretical predictions of Hanin and Nica [22], although there has yet to be a careful empirical comparison made between the precise predictions coming from infinite-depth-and-width models and real world networks.…”

Deep Learning Performance Characterization on GPUs for Various Quantization Frameworks

“…Quantization techniques have demonstrated remarkable performance improvements in training and inference of DNN models [35][36][37][38][39]. Similarly, the advancements in halfprecision and mixed-precision training [40,41] have played an important role in efficient DNN execution.…”