Accelerating Distributed Deep Learning By Adaptive Gradient Quantization

Guo, Jiong; Liu, Wantao; Wang, Wang; Han, Jizhong; Li, Ruixuan; Lu, Yijun; Hu, Songlin

doi:10.1109/icassp40776.2020.9054164

Cited by 16 publications

(8 citation statements)

References 2 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We use test accuracy to measure the learning performance. We compare our proposed DQ-SGD with the following baselines: SignSGD [5], TernGrad [12], QSGD [2], Adaptive [11] and AdaQS [9].…”

Section: Methodsmentioning

confidence: 99%

“…Several studies try to construct adaptive quantization schemes through engineering heuristics or empirical observations. However, they do not come up with a solid theoretical analysis [9], [10], [11], which even results in contradicted conclusions. More specifically, MQGrad [10], and AdaQS [11] suggest using few quantization bits in early epochs and gradually increase the number of bits in later epochs; while the scheme proposed by Anders [9] states that more quantization bits should be used for the gradient with a larger root-mean-squared (RMS) value, choosing to use more bits in the early training stage and fewer bits in the later stage.…”

Section: Related Workmentioning

confidence: 99%

“…However, they do not come up with a solid theoretical analysis [9], [10], [11], which even results in contradicted conclusions. More specifically, MQGrad [10], and AdaQS [11] suggest using few quantization bits in early epochs and gradually increase the number of bits in later epochs; while the scheme proposed by Anders [9] states that more quantization bits should be used for the gradient with a larger root-mean-squared (RMS) value, choosing to use more bits in the early training stage and fewer bits in the later stage. This paper's key contribution is to develop a systematic framework to crystallize the design trade-off in dynamic gradient quantization and settle this contradiction.…”

Section: Related Workmentioning

confidence: 99%

“…Existing algorithms often quantize parameters into a fixed number of bits for all training iterations, which is inefficient in balancing the communication-convergence tradeoff. To further reduce the communication overhead, some empirical studies began to dynamically adjust the number of quantization bits according to current model parameters in the training process, such as the gradient's mean to standard deviation ratio [9], the training loss [10], gradient's rootmean-squared value [11]. Though these empirical heuristics of adaptive quantization methods show good performance in specific tasks, their imprecise conjectures and the lack of theoretical guidelines in the conjecture framework have limited their generalization to a broad range of machine learning models/tasks.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

DQ-SGD: Dynamic Quantization in SGD for Communication-Efficient Distributed Learning

Yan¹,

Huang²,

Lan³

et al. 2021

Preprint

View full text Add to dashboard Cite

Gradient quantization is an emerging technique in reducing communication costs in distributed learning. Existing gradient quantization algorithms often rely on engineering heuristics or empirical observations, lacking a systematic approach to dynamically quantize gradients. This paper addresses this issue by proposing a novel dynamically quantized SGD (DQ-SGD) framework, enabling us to dynamically adjust the quantization scheme for each gradient descent step by exploring the trade-off between communication cost and convergence error. We derive an upper bound, tight in some cases, of the convergence error for a restricted family of quantization schemes and loss functions. We design our DQ-SGD algorithm via minimizing the communication cost under the convergence error constraints. Finally, through extensive experiments on large-scale natural language processing and computer vision tasks on AG-News, CIFAR-10, and CIFAR-100 datasets, we demonstrate that our quantization scheme achieves better tradeoffs between the communication cost and learning performance than other state-of-the-art gradient quantization methods.

show abstract

“…We use test accuracy to measure the learning performance. We compare our proposed DQ-SGD with the following baselines: SignSGD [5], TernGrad [12], QSGD [2], Adaptive [11] and AdaQS [9].…”

Section: Methodsmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

DQ-SGD: Dynamic Quantization in SGD for Communication-Efficient Distributed Learning

Yan¹,

Huang²,

Lan³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…In order to solve the problem of low model accuracy or low compression rate in the existing quantization schemes, Guo et al [ 11 ] proposed a novel adaptive quantization scheme(AdaQS). AdaQS algorithm according to the ratio of the mean and standard deviation of the gradient (MSDR) automatically determine the quantitative level, and achieve a balance between model accuracy and quantitative level.…”

Section: Related Workmentioning

confidence: 99%

A Partition Based Gradient Compression Algorithm for Distributed Training in AIoT

Guo

Liu

Zhang

2021

Sensors

View full text Add to dashboard Cite

Running Deep Neural Networks (DNNs) in distributed Internet of Things (IoT) nodes is a promising scheme to enhance the performance of IoT systems. However, due to the limited computing and communication resources of the IoT nodes, the communication efficiency of the distributed DNN training strategy is a problem demanding a prompt solution. In this paper, an adaptive compression strategy based on gradient partition is proposed to solve the problem of high communication overhead between nodes during the distributed training procedure. Firstly, a neural network is trained to predict the gradient distribution of its parameters. According to the distribution characteristics of the gradient, the gradient is divided into the key region and the sparse region. At the same time, combined with the information entropy of gradient distribution, a reasonable threshold is selected to filter the gradient value in the partition, and only the gradient value greater than the threshold is transmitted and updated, to reduce the traffic and improve the distributed training efficiency. The strategy uses gradient sparsity to achieve the maximum compression ratio of 37.1 times, which improves the training efficiency to a certain extent.

show abstract

Improving maize yield prediction at the county level from 2002 to 2015 in China using a novel deep learning approach

Geng

Zhang

et al. 2022

Computers and Electronics in Agriculture

View full text Add to dashboard Cite

Accelerating Distributed Deep Learning By Adaptive Gradient Quantization

Cited by 16 publications

References 2 publications

DQ-SGD: Dynamic Quantization in SGD for Communication-Efficient Distributed Learning

DQ-SGD: Dynamic Quantization in SGD for Communication-Efficient Distributed Learning

A Partition Based Gradient Compression Algorithm for Distributed Training in AIoT

Improving maize yield prediction at the county level from 2002 to 2015 in China using a novel deep learning approach

Contact Info

Product

Resources

About