FracTrain: Fractionally Squeezing Bit Savings Both Temporally and Spatially for Efficient DNN Training

Abstract

Recent breakthroughs in deep neural networks (DNNs) have fueled a tremendous demand for intelligent edge devices featuring on-site learning, while the practical realization of such systems remains a challenge due to the limited resources available at the edge and the required massive training costs for state-of-the-art (SOTA) DNNs. As reducing precision is one of the most effective knobs for boosting training time/energy efficiency, there has been a growing interest in low-precision DNN training. In this paper, we explore from an orthogonal direction: how to fractionally squeeze out more training cost savings from the most redundant bit level, progressively along the training trajectory and dynamically per input. Specifically, we propose FracTrain that integrates (i) progressive fractional quantization which gradually increases the precision of activations, weights, and gradients that will not reach the precision of SOTA static quantized DNN training until the final training stage, and (ii) dynamic fractional quantization which assigns precisions to both the activations and gradients of each layer in an input-adaptive manner, for only "fractionally" updating layer parameters. Extensive simulations and ablation studies (six models, four datasets, and three training settings including standard, adaptation, and fine-tuning) validate the effectiveness of FracTrain in reducing computational cost and hardware-quantified energy/latency of DNN training while achieving a comparable or better (-0.12% ∼ +1.87%) accuracy. For example, when training ResNet-74 on CIFAR-10, FracTrain achieves 77.6% and 53.5% computational cost and training latency savings, respectively, compared with the best SOTA baseline, while achieving a comparable (-0.07%) accuracy. Our codes are available at: https://github.com/RICE-EIC/FracTrain.Recent breakthroughs in deep neural networks (DNNs) have motivated an explosive demand for intelligent edge devices. Many of them, such as autonomous vehicles and healthcare wearables, require real-time and on-site learning to enable them to proactively learn from new data and adapt to dynamic environments. The challenge for such on-site learning is that the massive and growing cost of state-of-the-art (SOTA) DNNs stands at odds with the limited resources available at the edge devices, raising a major concern even when training in cloud using powerful GPUs/CPUs [1, 2]. To address the above challenge towards efficient DNN training, low-precision training have been developed recognizing that the training time/energy efficiency is a quadratic function of DNNs' adopted precision [3]. While they have showed promising training efficiency, they all adopt (i) a static quantization strategy, i.e., the precisions are fixed during the whole training process; (ii) 34th Conference on Neural Information Processing Systems (NeurIPS 2020),

Shift-BNN: Highly-Efficient Probabilistic Bayesian Neural Network Training via Memory-Friendly Pattern Retrieving

“…The significant reduction of DRAM access is the major source of Shift-BNN's high energy efficiency. As various lowerprecision training techniques [4,23,62] Scalability to larger sample size. In some high-risk applications, one may need a more robust BNN model to make decisions, thus requires training BNNs with a larger sample size to strictly approximate the loss function in Eq.1.…”

“…As the unit energy cost (J/bit) of off-chip memory accesses is orders of magnitude higher than that of MACs [11,15,30], data movement usually poses greater challenges for energy-efficient DNN training [56]. Moreover, the ongoing development of lowprecision training techniques [23,27,55] can potentially reduce the unit energy cost of MACs, but this could also result in a proportionally higher impact on the overall training's energy efficiency from the data movement. 1 (a).…”

F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantization

“…One area uses uniform-precision quantization where the model shares the same precision Choukroun et al, 2019;Gong et al, 2019;Langroudi et al, 2019;Jin et al, 2020a;Bhalgat et al, 2020;Darvish Rouhani et al, 2020;Oh et al, 2021). Another direction studies mixed-precision that determines bit-width for each layer through search algorithms, aiming at better accuracy-efficiency trade-off (Dong et al, 2019;Wang et al, 2019;Habi et al, 2020;Fu et al, 2020;Yang & Jin, 2020;Zhao et al, 2021a;b;Ma et al, 2021b). There is also binarization network, which only applies 1-bit (Rastegari et al, 2016;Hubara et al, 2016;Cai et al, 2017;Bulat et al, 2020;.…”

LDP: Learnable Dynamic Precision for Efficient Deep Neural Network Training and Inference

“…While various low precision training techniques have been proposed to boost DNNs' training efficiency [4,30,42,50], most of these techniques adopt a fixed precision allocation strategy throughout the whole training process, leaving a large room for further squeezing out bit-wise savings. Motivated by the recent pioneering works, which advocate that (1) different DNN layers behave differently through the training process [34,45] and (2) different DNN training stages favor different training schemes [27], a few pioneering works [10,11,24] have proposed to adopt dynamic training precision, which varies the precision spatially (e.g., layer-wise precision allocation) and temporally (e.g., different precision in different training epochs) and shows promising training efficiency and optimality over their static counterparts. However, existing dynamic low precision training methods rely on manually designed dynamic precision schedules [10,11,24], thus making it challenging to be directly applied to new models/tasks and limiting their achievable training efficiency.…”

“…) dynamic low precision training, (PFQ [11] and (3) CPT [10]). Here, each row shows the precision schedule for the whole model of the baselines, where LDP adopts a learned layer-wise dynamic precision schedule to optimally balance the training efficiency and accuracy trade-off; and (b) LDP's learned spatial precision distribution and temporal precision schedule for ResNet-38@CIFAR-100, where different curves correspond to spatial precision distributions of different residual blocks andthe fractional precision is due to the block-wise average and moving average among iterations for better visualization.…”