Collective Algorithms for Multiported Torus Networks

Sack, Paul; Gropp, William

doi:10.1145/2686882

Cited by 10 publications

(2 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…All-Reduce is a family of collective operations that allow nodes to efficiently aggregate (e.g. average) their local vectors and distribute the result across all devices [51,52,53]. Unlike parameter servers, All-Reduce assigns equal roles to all devices, making it easier to scale to a large number of homogeneous workers.…”

Section: Distributed Trainingmentioning

confidence: 99%

Distributed Deep Learning in Open Collaborations

Diskin¹,

Bukhtiyarov²,

Ryabinin³

et al. 2021

Preprint

View full text Add to dashboard Cite

Modern deep learning applications require increasingly more compute to train state-of-the-art models. To address this demand, large corporations and institutions use dedicated High-Performance Computing clusters, whose construction and maintenance are both environmentally costly and well beyond the budget of most organizations. As a result, some research directions become the exclusive domain of a few large industrial and even fewer academic actors. To alleviate this disparity, smaller groups may pool their computational resources and run collaborative experiments that benefit all participants. This paradigm, known as grid-or volunteer computing, has seen successful applications in numerous scientific areas. However, using this approach for machine learning is difficult due to high latency, asymmetric bandwidth, and several challenges unique to volunteer computing. In this work, we carefully analyze these constraints and propose a novel algorithmic framework designed specifically for collaborative training. We demonstrate the effectiveness of our approach for SwAV and ALBERT pretraining in realistic conditions and achieve performance comparable to traditional setups at a fraction of the cost. Finally, we provide a detailed report of successful collaborative language model pretraining with 40 participants.

show abstract

Section: Distributed Trainingmentioning

confidence: 99%

Distributed Deep Learning in Open Collaborations

Diskin¹,

Bukhtiyarov²,

Ryabinin³

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…If N =M d and there are no node/network failures, Algorithm 1 is equivalent to Torus All-Reduce (Sack & Gropp, 2015), achieving the exact average after d rounds of communication (see Appendix C.1). However, our typical use case is far from this perfect scenario; for example, some groups can have less than M members.…”

Section: Moshpit Averagingmentioning

confidence: 99%

Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices

Ryabinin¹,

Gorbunov²,

Plokhotnyuk³

et al. 2021

Preprint

View full text Add to dashboard Cite

Training deep neural networks on large datasets can often be accelerated by using multiple compute nodes. This approach, known as distributed training, can utilize hundreds of computers via specialized message-passing protocols such as Ring All-Reduce. However, running these protocols at scale requires reliable high-speed networking that is only available in dedicated clusters. In contrast, many real-world applications, such as federated learning and cloud-based distributed training, operate on unreliable devices with unstable network bandwidth. As a result, these applications are restricted to using parameter servers or gossip-based averaging protocols. In this work, we lift that restriction by proposing Moshpit All-Reduce -an iterative averaging protocol that exponentially converges to the global average. We demonstrate the efficiency of our protocol for distributed optimization with strong theoretical guarantees. The experiments show 1.3× speedup for ResNet-50 training on ImageNet compared to competitive gossip-based strategies and 1.5× speedup when training ALBERT-large from scratch using preemptible compute nodes.Running large-scale distributed training in these circumstances requires fault-and latency-tolerant algorithms (Lian et al., 2017;Assran et al., 2019b). Most of these algorithms replace all-reduce averaging with gossip: each participant periodically downloads the latest parameters from his neigh-

show abstract