Prague: High-Performance Heterogeneity-Aware Asynchronous Decentralized Training

Abstract

Distributed deep learning training usually adopts All-Reduce as the synchronization mechanism for data parallel algorithms due to its high performance in homogeneous environment. However, its performance is bounded by the slowest worker among all workers, and is significantly slower in heterogeneous situations. AD-PSGD, a newly proposed synchronization method which provides numerically fast convergence and heterogeneity tolerance, suffers from deadlock issues and high synchronization overhead. Is it possible to get the best of both worlds -designing a distributed training method that has both high performance as All-Reduce in homogeneous environment and good heterogeneity tolerance as AD-PSGD?In this paper, we propose Ripples, a high-performance heterogeneity-aware asynchronous decentralized training approach. We achieve the above goal with intensive synchronization optimization, emphasizing the interplay between algorithm and system implementation. To reduce synchronization cost, we propose a novel communication primitive Partial All-Reduce that allows a large group of workers to synchronize quickly. To reduce synchronization conflict, we propose static group scheduling in homogeneous environment and simple techniques (Group Buffer and Group Division) to avoid conflicts with slightly reduced randomness. Our experiments show that in homogeneous environment, Ripples is 1.1× faster than the state-of-the-art implementation of All-Reduce, 5.1× faster than Parameter Server and 4.3× faster than AD-PSGD. In a heterogeneous setting, Ripples shows 2× speedup over All-Reduce, and still obtains 3× speedup over the Parameter Server baseline.

Elastic parameter server load distribution in deep learning clusters

“…Some advocate less strict parameter synchronization to mitigate the synchronization cost, such as Stale Synchronous Parallel (SSP) [28], Dynamic Synchronous Parallel [48] and Round-Robin Synchronous Parallel [15]. AD-PSGD [29] and Hop [30] are variations of asynchronous and stale synchronous training, which target communication efficiency in heterogeneous environments. Some other studies focus on heterogeneity-aware distributed SGD algorithms, employing a constant learning rate for delayed gradient push in the SSP protocol, to reduce disturbance and unstable convergence caused by stragglers [25].…”

“…AllReduce methods [12,33,34] also lack the flexibility to tackle straggler issues, which is more challenging than PS architecture due to the more restrictive communication pattern between workers. There are some works focusing on straggler issues in AllReduce [29,30], but their methods may lead to deadlocks and may not be able to deal with complex straggler patterns, such as transient stragglers.…”

Mitigating Stragglers in the Decentralized Training on Heterogeneous Clusters

“…Even though the slow workers inevitably have staler parameters, the e ects on the training of the global model can be minimized via the probabilistic solution. However, this strategy requires additional system overhead and manual e orts to generate a dynamic communication graph [39].…”

“…Several pieces of research have explored the robustness of deep learning processes [37] [39]. In particular, AD-PSGD [37] is the rst that proposes to use randomized communication to reduce the e ects of stragglers probabilistically.…”

“…However, this design incurs signi cant synchronization overhead to ensure atomicity. It also requires manual e orts to avoid scheduling con icts [39]. Inspired by the fact that deep learning training process is robust to bounded errors, we propose to relax the global barrier without changing the communication graph to mitigate the impact from "long-tail e ects. "…”