Effect of dynamic algorithm selection of Alltoall communication on environments with unstable network speed

Abstract

As the HPC systems increase their size, performance of collective communications is becoming an important issue. Usually, decisions for which algorithm of those communications to be used are done based on statically specified thresholds of the size of messages and the number of processes. However, on recent HPC systems that are hiring Fat Tree or Torus topology as their interconnect, the network speed has become unpredictable. The main reason is the effect of contentions. This effect depends heavily on the relative locations of the compute nodes. On the other hand, to reduce the number of idle nodes, there are attempts for building job schedulers to attach compute nodes flexibly, without considering their relative positions among each other. With this policy, the network performance becomes unstable. As an approach for finding an appropriate algorithm even on such environment, a dynamic method, STAR-MPI, has been proposed. This method examines each algorithm at runtime, and uses the empirical data to choose the suitable one for the given situation.This paper first examined the effect of STAR-MPI on an environment with unstable network speed. The results of experiments on this environment showed that the dynamic approach was effective, but the cost for testing slow algorithms limited the effect. Then, the authors proposed an enhancement, in which algorithms that have been predicted relatively slow were discarded from the list of candidates. The predictions were done by using the performance models of the algorithms with the latency and the bandwidth measured at the first call of the collective communication. At this point, the effect of this enhancement shown in experimental results was not significant. However, the results showed that there was a possibility for achieving better performance by using more cost-effective way of prediction and tuning thresholds and factors used in the enhancement.

Communication Optimization Technology Based on Network Dynamic Performance Model

“…As for the supercomputing system, after the system reaches a certain scale, the delay, bandwidth, and blocking of the communication between nodes are greatly affected by the network topology. In order to achieve the reasonable map between data distribution dimension and the system network topology, it is necessary to detect system data communication dynamic topology, through test sets and test system (including nodes between [10,11] designed a method called star-MPI (self-tuning adaptive routines for MPI collective operations), which can dynamically select the algorithm for ensemble communication in a network with unpredictable performance. is method tests various possible schemes and uses a certain prediction mechanism to delete the algorithm with low performance to save testing time.…”

Performance Analysis of Message Passing Interface Collective Communication on Intel Xeon Quad-Core Gigabit Ethernet and Infiniband Clusters

“…Other studies provide performance analysis of point to point or collective communication on different interconnects (Ismail et al, 2011;Rashti and Afsahi, 2007) while some provide comparison and analysis of multiple algorithms for collective communication in order to find the best solution for different parallel systems (Nanri and Kurokawa, 2011;Hamid and Coddington, 2007). Other related studies focused on optimizing the performance of MPI collective communication by proposing topology aware mechanisms (Gong et al, 2013;Subramoni et al, 2011;Kandalla et al, 2010) and process arrival patterns aware mechanisms (Qian and Afsahi, 2009;Patarasuk and Yuan, 2008) to achieve the best performance in terms of time.…”