SALaR: Scalable and Adaptive Designs for Large Message Reduction Collectives

Bayatpour, Mohammadreza; Hashmi, Jahanzeb Maqbool; Chakraborty, Sourav; Subramoni, Hari; Kousha, Pouya; Panda, Dhabaleswar K.

doi:10.1109/cluster.2018.00014

Cited by 24 publications

(5 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…For the synchronous data‐parallel approaches, major frameworks and GPU‐based libraries optimize the communication scheme for such large message size, ie, a bandwidth‐optimal algorithm for Allreduce operation. The work by Bayatpour et al 42 summarized the state‐of‐the‐art designs and features of MPI AllReduce in the literature for large messages that were covered in our evaluation. For example, Baidu‐AllReduce and Segmented Ring implemented the logical ring‐based algorithm.…”

Section: Discussion and Related Workmentioning

confidence: 99%

“…It is reported that those algorithms can be applied to GPU systems, which is the main target in this work. By contrast, XPMEM‐based Reduction and SALaR algorithms 42 are proposed to support systems with CPU only, ie, based on the shared‐memory architecture. We, hence, do not consider those algorithms for comparison with our proposal.…”

Section: Discussion and Related Workmentioning

confidence: 99%

See 1 more Smart Citation

Efficient MPI‐AllReduce for large‐scale deep learning on GPU‐clusters

Nguyen

Wahib

Takano

2019

Concurrency and Computation

View full text Add to dashboard Cite

Training models on large-scale GPUs-accelerated clusters are becoming a commonplace due to the increase in complexity and size in Deep Learning models. One of the main challenges for distributed training is the collective communication overhead for large message sizes: up to hundreds of MB. In this paper, we propose two hierarchical distributed memory multi-leader allreduce algorithms optimized for GPU-accelerated clusters (named lr_lr and lr_rab). In which, GPUs inside a computing node perform an intra-node communication phase to gather and store results of local reduced values to designated GPUs (known as node leaders). Node leaders then keep a role as an inter-node communicator. Each leader exchanges one part of reduced values to the leaders of the other nodes in parallel. Hence, we are capable of significantly reducing the time for injecting data into the inter-node network. We also overlap the inter-node and intra-node communication by implementing our proposal in a pipelined manner. We evaluate those algorithms on the discrete-event simulation Simgrid. We show that our algorithms, lr_lr and lr_rab, can cut down the execution time of an Allreduce micro-benchmark that uses the logical ring algorithm (lr) by up to 45% and 51%, respectively. With the pipelined implementation, our lr_lr_pipe achieve 15% performance improvement when compared to lr_lr. In addition, simulation result also project power savings for the network devices of up to 23% and 32%.

show abstract

Section: Discussion and Related Workmentioning

confidence: 99%

Section: Discussion and Related Workmentioning

confidence: 99%

Efficient MPI‐AllReduce for large‐scale deep learning on GPU‐clusters

Nguyen

Wahib

Takano

2019

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…Another collective that employs a ring algorithm for large messages is Allreduce. This collective has been heavily studied and continuously gets improved by the academic community [12][13][14] ; it is frequently used in both traditional HPC and DL operations.…”

Section: Motivationmentioning

confidence: 99%

Accelerating communication with multi‐HCA aware collectives in MPI

Tran,

Ramesh,

Michalowicz

et al. 2023

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

SummaryTo accelerate the communication between nodes, supercomputers are now equipped with multiple network adapters per node, also referred to as HCAs (Host Channel Adapters), resulting in a “multi‐rail”/“multi‐HCA” network. For example, the ThetaGPU system at Argonne National Laboratory (ANL) has eight adapters per node; with this many networking resources available, utilizing all of them becomes non‐trivial. The Message Passing Interface (MPI) is a dominant model for high‐performance computing clusters. Not all MPI collectives utilize all resources, and this becomes more apparent with advances in bandwidth and adapter count in a given cluster. In this work, we provide a thorough performance analysis of existing multirail solutions and their implications on collectives and present the necessity for further enhancement. Specifically, we propose novel designs for hierarchical, multi‐HCA‐aware Allgather. The proposed designs fully utilize all the available network adapters within a node and provide high overlap between inter‐node and intra‐node communication. At the micro‐benchmark level, we see large inter‐node improvements up to 62% and 61% better than HPC‐X and MVAPICH2‐X for 1024 processes. Because Allgather is used in Ring‐Allreduce, our designs also improve its performance by 56% and 44% compared to HPC‐X and MVAPICH2‐X, respectively. At the application level, our enhanced Allgather shows and improvement in a matrix‐vector multiplication kernel when compared to HPC‐X and MVAPICH2‐X, and Allreduce performs up to 7.83% better in deep learning training against MVAPICH2‐X.

show abstract

“…The approach works well for messages up to 256 Kbytes. The mechanism is improved later for larger message sizes by introducing pipelining [4].…”

Section: Related Workmentioning

confidence: 99%

Analyzing the performance of hierarchical collective algorithms on ARM-based multicore clusters

Utrera¹,

Gil²,

Martorell³

2022

2022 30th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP)

View full text Add to dashboard Cite

MPI is the de facto communication standard library for parallel applications in distributed memory architectures. Collective operations performance is critical in HPC applications as they can become the bottleneck of their executions. The advent of larger node sizes on multicore clusters has motivated the exploration of hierarchical collective algorithms aware of the process placement in the cluster and the memory hierarchy. This work analyses and compares several hierarchical collective algorithms from the literature that do not form part of the current MPI standard. We implement the algorithms on top of OpenMPI using the shared-memory facility provided by MPI-3 at the intra-node level and evaluate them on ARM-based multicore clusters. From our results, we evidence aspects of the algorithms that impact the performance and applicability of the different algorithms. Finally, we propose a model that helps us to analyze the scalability of the algorithms.

show abstract

SALaR: Scalable and Adaptive Designs for Large Message Reduction Collectives

Cited by 24 publications

References 18 publications

Efficient MPI‐AllReduce for large‐scale deep learning on GPU‐clusters

Efficient MPI‐AllReduce for large‐scale deep learning on GPU‐clusters

Accelerating communication with multi‐HCA aware collectives in MPI

Analyzing the performance of hierarchical collective algorithms on ARM-based multicore clusters

Contact Info

Product

Resources

About