Decomposing MPI Collectives for Exploiting Multi-lane Communication

Träff, Jesper Larsson; Hunold, Sascha

doi:10.1109/cluster49012.2020.00037

Cited by 17 publications

(8 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Our first results seem to confirm that hierarchical algorithms that are aware of the underlying platform are needed. We confirm that the currently available MPI libraries demonstrate severe problems in that sense [5]. However, the hierarchical algorithm needs to adapt to parameters other than the size of nodes, like memory hierarchy and interconnection networks.…”

Section: Introductionsupporting

confidence: 71%

“…The work presented by [5] indicates considerable room for improvement with the MPI collectives in current libraries, using multi-lane communication. Zhou et al [8] demonstrate that the use of shared-memory windows introduced by the MPI-3 standard may improve the performance of the MPI collective operations.…”

Section: Related Workmentioning

confidence: 99%

“…2) Multi-lane. This algorithm proposed by J. Traff et al [5] suggests having multiple lanes per group. The message is partitioned and scattered among the processes of the group.…”

Section: A Hierarchical Algorithms Evaluatedmentioning

confidence: 99%

“…3) Hier. This algorithm is another alternative to Multilane proposed in the work by J. Traff et al [5] as well. It is very similar to Multi-leader, but without using explicitly shared memory.…”

Section: A Hierarchical Algorithms Evaluatedmentioning

confidence: 99%

“…There are some proposals done in the area which have not been included yet in the MPI standard [3][4] [5]. The design of the algorithms is aware of the process placement in the cluster and try to take advantage of shared memory and the available parallelism within a node.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Analyzing the performance of hierarchical collective algorithms on ARM-based multicore clusters

Utrera¹,

Gil²,

Martorell³

2022

2022 30th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP)

View full text Add to dashboard Cite

MPI is the de facto communication standard library for parallel applications in distributed memory architectures. Collective operations performance is critical in HPC applications as they can become the bottleneck of their executions. The advent of larger node sizes on multicore clusters has motivated the exploration of hierarchical collective algorithms aware of the process placement in the cluster and the memory hierarchy. This work analyses and compares several hierarchical collective algorithms from the literature that do not form part of the current MPI standard. We implement the algorithms on top of OpenMPI using the shared-memory facility provided by MPI-3 at the intra-node level and evaluate them on ARM-based multicore clusters. From our results, we evidence aspects of the algorithms that impact the performance and applicability of the different algorithms. Finally, we propose a model that helps us to analyze the scalability of the algorithms.

show abstract

Section: Introductionsupporting

confidence: 71%

Section: Related Workmentioning

confidence: 99%

“…2) Multi-lane. This algorithm proposed by J. Traff et al [5] suggests having multiple lanes per group. The message is partitioned and scattered among the processes of the group.…”

Section: A Hierarchical Algorithms Evaluatedmentioning

confidence: 99%

Section: A Hierarchical Algorithms Evaluatedmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Analyzing the performance of hierarchical collective algorithms on ARM-based multicore clusters

Utrera¹,

Gil²,

Martorell³

2022

2022 30th Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP)

View full text Add to dashboard Cite

show abstract

Accelerating communication with multi‐HCA aware collectives in MPI

Tran,

Ramesh,

Michalowicz

et al. 2023

Concurrency and Computation

View full text Add to dashboard Cite

SummaryTo accelerate the communication between nodes, supercomputers are now equipped with multiple network adapters per node, also referred to as HCAs (Host Channel Adapters), resulting in a “multi‐rail”/“multi‐HCA” network. For example, the ThetaGPU system at Argonne National Laboratory (ANL) has eight adapters per node; with this many networking resources available, utilizing all of them becomes non‐trivial. The Message Passing Interface (MPI) is a dominant model for high‐performance computing clusters. Not all MPI collectives utilize all resources, and this becomes more apparent with advances in bandwidth and adapter count in a given cluster. In this work, we provide a thorough performance analysis of existing multirail solutions and their implications on collectives and present the necessity for further enhancement. Specifically, we propose novel designs for hierarchical, multi‐HCA‐aware Allgather. The proposed designs fully utilize all the available network adapters within a node and provide high overlap between inter‐node and intra‐node communication. At the micro‐benchmark level, we see large inter‐node improvements up to 62% and 61% better than HPC‐X and MVAPICH2‐X for 1024 processes. Because Allgather is used in Ring‐Allreduce, our designs also improve its performance by 56% and 44% compared to HPC‐X and MVAPICH2‐X, respectively. At the application level, our enhanced Allgather shows and improvement in a matrix‐vector multiplication kernel when compared to HPC‐X and MVAPICH2‐X, and Allreduce performs up to 7.83% better in deep learning training against MVAPICH2‐X.

show abstract