Hierarchical redesign of classic MPI reduction algorithms

Hasanov, Khalid; Lastovetsky, Alexey

doi:10.1007/s11227-016-1779-7

Cited by 25 publications

(12 citation statements)

References 17 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The MPI instances usually optimize the CCPs via the implementation of a variety of algorithms (or communication schemes), in principle selecting the most appropriate option at execution time depending, for example, on the message size, number of processes, network topology, etc. For the particular case of (the non-blocking) MPI_Iallreduce, the following list briefly describes some of the most popular algorithms (see [9,12,30] for additional details):…”

Section: A Family Of Algorithmsmentioning

confidence: 99%

Accelerating distributed deep neural network training with pipelined MPI allreduce

2021

View full text Add to dashboard Cite

TensorFlow (TF) is usually combined with the Horovod (HVD) workload distribution package to obtain a parallel tool to train deep neural network on clusters of computers. HVD in turn utilizes a blocking Allreduce primitive to share information among processes, combined with a communication thread to overlap communication with computation. In this work, we perform a thorough experimental analysis to expose (1) the importance of selecting the best algorithm in MPI libraries to realize the Allreduce operation; and (2) the performance acceleration that can be attained when replacing a blocking Allreduce with its non-blocking counterpart (while maintaining the blocking behaviour via the appropriate synchronization mechanism). Furthermore, (3) we explore the benefits of applying pipelining to the communication exchange, demonstrating that these improvements carry over to distributed training via TF+HVD. Finally, (4) we show that pipelining can also boost performance for applications that make heavy use of other collectives, such as Broadcast and Reduce-Scatter.

show abstract

Section: A Family Of Algorithmsmentioning

confidence: 99%

Accelerating distributed deep neural network training with pipelined MPI allreduce

2021

View full text Add to dashboard Cite

show abstract

“…Existing analytical modelling approaches [5], [23], [40] estimate the execution time of the binary and binomial tree broadcast algorithms as follows:…”

Section: Comparison Of the Proposed Analytical Models Against The State Of The Artmentioning

confidence: 99%

Efficient and Accurate Selection of Optimal Collective Communication Algorithms Using Analytical Performance Modeling

Nuriyev

Lastovetsky

2021

IEEE Access

Self Cite

View full text Add to dashboard Cite

The performance of collective operations has been a critical issue since the advent of Message Passing Interface (MPI). Many algorithms have been proposed for each MPI collective operation but none of them proved optimal in all situations. Different algorithms demonstrate superior performance depending on the platform, the message size, the number of processes, etc. MPI implementations perform the selection of the collective algorithm empirically, executing a simple runtime decision function. While efficient, this approach does not guarantee the optimal selection. As a more accurate but equally efficient alternative, the use of analytical performance models of collective algorithms for the selection process was proposed and studied. Unfortunately, the previous attempts in this direction have not been successful. We revisit the analytical model-based approach and propose two innovations that significantly improve the selective accuracy of analytical models: (1) We derive analytical models from the code implementing the algorithms rather than from their high-level mathematical definitions. This results in more detailed models.(2) We estimate model parameters separately for each collective algorithm and include the execution of this algorithm in the corresponding communication experiment. We experimentally demonstrate the accuracy and efficiency of our approach using Open MPI broadcast and gather algorithms and two different Grid'5000 clusters and one supercomputer.

show abstract

“…Recently, apart of the imbalanced PAPs subject [9], the focus of MPI reduce algorithms were mainly scoped on hierarchical optimization, where the processes are distributed over a compute cluster, and more than one process is assigned to the same node. In [7], Hasanov et al proposed an algorithm categorizing the processes into the local (in the same node) and global (distributed over a compute cluster) communicators, where the reduce can be performed in two phases, increasing the performance of the operation. In [21], Shan et al proposed utilization of idle threads on manycore processors and data compression to boost MPI reduce performance, and in [28], Zhao et al showed that usage of k-nomial tree algorithms has the advantage over a typical binomial solution.…”

Section: Related Workmentioning

confidence: 99%

Improving Clairvoyant: reduction algorithm resilient to imbalanced process arrival patterns

Proficz

Ocetkiewicz

2020

J Supercomput

View full text Add to dashboard Cite

The Clairvoyant algorithm proposed in “A novel MPI reduction algorithm resilient to imbalances in process arrival times” was analyzed, commented and improved. The comments concern handling certain edge cases in the original pseudocode and description, i.e., adding another state of a process, improved cache friendliness more precise complexity estimations and some other issues improving the robustness of the algorithm implementation. The proposed improvements include skipping of idle loop rounds, simplifying generation of the ready set and management of the state array and an about 90-fold reduction in memory usage. Finally an extension enabling process arrival times (PATs) prediction was added: an additional background thread used to exchange the data with the PAT estimations. The performed tests, with a dedicated mini-benchmark executed in an HPC environment, showed correctness and improved performance of the solution, with comparison to the original or other state-of-the-art algorithms.

show abstract

Hierarchical redesign of classic MPI reduction algorithms

Cited by 25 publications

References 17 publications

Accelerating distributed deep neural network training with pipelined MPI allreduce

Accelerating distributed deep neural network training with pipelined MPI allreduce

Efficient and Accurate Selection of Optimal Collective Communication Algorithms Using Analytical Performance Modeling

Improving Clairvoyant: reduction algorithm resilient to imbalanced process arrival patterns

Contact Info

Product

Resources

About