Understanding Top-k Sparsification in Distributed Deep Learning

Abstract

Distributed stochastic gradient descent (SGD) algorithms are widely deployed in training large-scale deep learning models, while the communication overhead among workers becomes the new system bottleneck. Recently proposed gradient sparsification techniques, especially Top-k sparsification with error compensation (TopK-SGD), can significantly reduce the communication traffic without obvious impact on the model accuracy. Some theoretical studies have been carried out to analyze the convergence property of TopK-SGD. However, existing studies do not dive into the details of Top-k operator in gradient sparsification and use relaxed bounds (e.g., exact bound of Random-k) for analysis; hence the derived results cannot well describe the real convergence performance of TopK-SGD. To this end, we first study the gradient distributions of TopK-SGD during training process through extensive experiments. We then theoretically derive a tighter bound for the Top-k operator. Finally, we exploit the property of gradient distribution to propose an approximate top-k selection algorithm, which is computing-efficient for GPUs, to improve the scaling efficiency of TopK-SGD by significantly reducing the computing overhead. Codes are available at: https://github.com/hclhkbu/GaussianK-SGD.

Near-Optimal Sparse Allreduce for Distributed Deep Learning

“…Gradient sparsification [2,4,11,16,19,36,[41][42][43]50] is a key approach to lower the communication volume. By top-𝑘 selection, i.e., only selecting the largest (in terms of the absolute value) 𝑘 of 𝑛 components, the gradient becomes very sparse (commonly around 99%).…”

“…Then, the accumulated sparse gradient is used in the Stochastic Gradient Descent (SGD) optimizer to update the model parameters, which is called Top𝑘 SGD. The convergence of Top𝑘 SGD has been theoretically and empirically proved [4,36,41]. However, the parallel scalablity of the existing sparse allreduce algorithms is limited, which makes it very difficult to obtain real performance improvement, especially on the machines (e.g., supercomputers) with high-performance interconnected networks [5,17,37,40].…”

“…To solve the fill-in problem, gTop𝑘 hierarchically selects top-𝑘 values in each level of the reduction tree, which results in 4𝑘 log 𝑃 communication volume. Gaussian𝑘 [41] uses the same sparse allreduce algorithm as Top𝑘A with a further optimization for top-𝑘 selection. For our O𝑘-Top𝑘, the communication volume is bounded by 6𝑘.…”

“…Bitonic top-𝑘 [39] is a GPU-friendly algorithm with complexity O(𝑛 log 2 𝑘), but still not good enough for large 𝑘. To lower the overhead of sparsification, Gaussian𝑘 [41] approximates the gradient values distribution to a Gaussian distribution with the same mean and standard deviation, and then estimates a threshold using the percent-point function and selects the values above the threshold. The top-𝑘 selection in Gaussian𝑘 is GPU-friendly with complexity O(𝑛), but it usually underestimates the value of 𝑘 because of the difference between Gaussian and the real distributions (see Section 3.1.3).…”

“…Specifically, the communication volume of the existing sparse reduction algorithms grows with the number of processes 𝑃. Taking the allgather-based sparse reduction [36,41,47] as an example, its communication volume is proportional to 𝑃, which eventually surpasses the dense allreduce as 𝑃 increases. Other more complex algorithms [36] suffer from significant fill-in during the reduction, which also leads to a quick increase of the data volume as 𝑃 grows, and may degrade to dense representations on the fly.…”