Hierarchical Optimization of MPI Reduce Algorithms

Hierarchical redesign of classic MPI reduction algorithms

“…This means that the hierarchical transformation can improve the native reduction operation as well. It is expected that [7] the overhead from the MPI_Comm_split operation should affect only reduce operations with smaller message sizes. Figure 3 validates this with experimental results.…”

“…Figure 4 presents experiments with the hierarchical pipeline reduce with a message size of 16 KB with 1 KB segmentation. The performance of the pipeline algorithm with larger messages and segment sizes of 32 and 64 KB can be found in [7]. Figure 5 shows the speedup of the hierarchical transformation of native Open MPI reduce operation, linear, chain, pipeline, binary, binomial, and in-order binary reduce algorithms with message sizes starting from 16 KB up to 16 MB.…”

“…theoretical costs given in the Table 1. We do not provide theoretical analysis of those algorithms, as this work is an extension of our conference paper [7] which already provides detailed discussion of them. Instead, we provide a broader discussion of the MPI allreduce algorithms.…”

“…Because of the limited space, we only show theoretical analysis of the hierarchical flat tree reduce algorithm here. A detailed theoretical analysis of the other hierarchical reduce algorithms can be found in our previous work [7].…”

“…Hierarchical topology-oblivious transformation of existing communication algorithms has been recently proposed as a new promising approach to optimization of MPI collective communication algorithms and MPI-based applications [4,5,17]. This approach has been successfully applied to the most popular parallel matrix multiplication algorithm, SUMMA [6], and the state-of-the-art MPI broadcast and reduce algorithms, demonstrating significant multifold performance gains, especially on large-scale HPC systems [18]. In this article, we extend our conference paper [7] on topology-oblivious hierarchical optimization of MPI reduce operation by including a study of optimization of MPI allreduce operation using the same technique.…”