Designing Hierarchical Multi-HCA Aware Allgather in MPI

Tran, Tu; Michalowicz, Benjamin; Ramesh, Bharath; Subramoni, Hari; Shafi, Aamir; Panda, Dhabaleswar K.

doi:10.1145/3547276.3548524

Cited by 1 publication

(1 citation statement)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this paper, we take up these challenges and propose multi‐HCA aware designs. In addition to preliminary results from a recently published study, 16 we do the following in addition to them: (1) We first study the performance characterization of existing multirail solutions. (2) With the insights, we then propose novel multirail‐aware collectives.…”

Section: Introductionmentioning

confidence: 99%

Accelerating communication with multi‐HCA aware collectives in MPI

Tran,

Ramesh,

Michalowicz

et al. 2023

Concurrency and Computation

Self Cite

View full text Add to dashboard Cite

SummaryTo accelerate the communication between nodes, supercomputers are now equipped with multiple network adapters per node, also referred to as HCAs (Host Channel Adapters), resulting in a “multi‐rail”/“multi‐HCA” network. For example, the ThetaGPU system at Argonne National Laboratory (ANL) has eight adapters per node; with this many networking resources available, utilizing all of them becomes non‐trivial. The Message Passing Interface (MPI) is a dominant model for high‐performance computing clusters. Not all MPI collectives utilize all resources, and this becomes more apparent with advances in bandwidth and adapter count in a given cluster. In this work, we provide a thorough performance analysis of existing multirail solutions and their implications on collectives and present the necessity for further enhancement. Specifically, we propose novel designs for hierarchical, multi‐HCA‐aware Allgather. The proposed designs fully utilize all the available network adapters within a node and provide high overlap between inter‐node and intra‐node communication. At the micro‐benchmark level, we see large inter‐node improvements up to 62% and 61% better than HPC‐X and MVAPICH2‐X for 1024 processes. Because Allgather is used in Ring‐Allreduce, our designs also improve its performance by 56% and 44% compared to HPC‐X and MVAPICH2‐X, respectively. At the application level, our enhanced Allgather shows and improvement in a matrix‐vector multiplication kernel when compared to HPC‐X and MVAPICH2‐X, and Allreduce performs up to 7.83% better in deep learning training against MVAPICH2‐X.

show abstract

Section: Introductionmentioning

confidence: 99%