Automatic generation and tuning of MPI collective communication routines

Abstract

In order for collective communication routines to achieve high performance on different platforms, they must be able to adapt to the system architecture and use different algorithms for different situations. Current Message Passing Interface (MPI) implementations, such as MPICH and LAM/MPI, are not fully adaptable to the system architecture and are not able to achieve high performance on many platforms. In this paper, we present a system that produces efficient MPI collective communication routines. By automatically generating topology specific routines and using an empirical approach to select the best implementations, our system adapts to a given platform and constructs routines that are customized for the platform. The experimental results show that the tuned routines consistently achieve high performance on clusters with different network topologies.

Pipelined broadcast on Ethernet switched clusters

“…The results for topologies (2) and (3) are similar to those for topology (4). Since all algorithms run over MPICH except LAM, we also include a binomial tree implementation (the algorithm used in LAM) over MPICH in the comparison.…”

“…Figure 8 shows the performance of pipelined broadcast using different contention-free trees on topology (1). The performance of pipelined broadcast on topologies (2), (3), (4), and (5) has a similar trend. As can be seen from the figure, when the message size is large (≥ 32KB), the linear tree offers the best performance.…”

“…Since we only consider broadcasts with msize ≥ 8KB, the barrier overhead is insignificant to the total communication time. When reporting the performance of pipelined broadcast, by default, we only report the results with optimal segment sizes, which are determined by an automatic tuning system [4] using an empirical approach. Note that, as shown in Figure 12, while the optimal segment sizes are difficult to obtain, a wide range of segment sizes can yield performance close to the optimal.…”

Techniques for pipelined broadcast on ethernet switched clusters

“…Instead, we develop practical techniques to facilitate the deployment of pipelined broadcast on clusters connected by multiple Ethernet switches. Similar to other architecture specific collective communication algorithms [8,10,17], the techniques developed in this paper can be used in advanced communication libraries [7,9,13,30]. Our research extends the work in [11,23,28] by considering multiple switches.…”

“…There are techniques to develop adaptive MPI routines that use different algorithms according to the message sizes [7,20]. These adaptive techniques allow our algorithms and the complementary algorithms for broadcasting small messages to co-exist in one MPI routine.…”