Optimizing the steady-state throughput of scatter and reduce operations on heterogeneous platforms

Abstract

In this paper, we consider the communications involved by the execution of a complex application, deployed on a heterogeneous large-scale distributed platform. Such applications intensively use collective macro-communication schemes, such as scatters, personalized all-to-alls or gather/reduce operations. Rather than aiming at minimizing the execution time of a single macro-communication, we focus on the steady-state operation. We assume that there is a large number of macro-communications to perform in pipeline fashion, and we aim at maximizing the throughput, i.e., the (rational) number of macro-communications which can be initiated every time-step. We target heterogeneous platforms, modeled by a graph where resources have different communication and computation speeds. The situation is simpler for series of scatters or personalized all-to-alls than for series of reduces operations, because of the possibility of combining various partial reductions of the local values, and of interleaving computations with communications. In all cases, we show how to determine the optimal throughput, and how to exhibit a concrete periodic schedule that achieves this throughput.

Scheduling associative reductions with homogeneous costs when overlapping communications and computations

“…Legrand, Marchal and Robert [11] study steady-state situations where a series of reductions are performed. As in our work, the reduction operation is assumed to be indivisible, transfers and computations can overlap and the full-duplex one-port model is considered.…”

Complexity results and heuristics for pipelined multicast operations on heterogeneous platforms

“…But the general problem is very difficult: we have proved that determining the optimal throughput is a NP-complete problem. This negative result demonstrates that pipelining multicasts is more difficult than pipelining broadcasts, scatters or reduce operations, for which optimal polynomial algorithms have been introduced [6,22]. In particular, although the broadcast and the multicast problems seem very similar (the target set is the only difference), there is a complexity gap between them: the best throughput to broadcast message to all nodes in the platform can be found in polynomial time, whereas finding the best throughput to multicast a message to a strict subset of these nodes is NP-hard.…”

“…From the solution of the Multicast-UB(P, P target ) linear program, a solution achieving the same throughput can be easily obtained. The interested reader may refer to [22,21] where the reconstruction scheme from the linear program is presented. In this section, we prove that such a solution may differ at most by a factor |P target | from the solution of the throughput given by the solution of Multicast-LB(P, P target ), thus providing a heuristic with a guaranteed factor |P target |.…”

“…In previous papers, we have dealt with other communication primitives than the multicast operation. We have shown how to compute the optimal steady-state throughput for a series of scatter or reduce operations [22,21], and a series of broadcast operations [6,5]. The idea is to characterize the steady-state operation of each resource through a linear program in rational numbers (that can thus be solved with a complexity polynomial in the platform size), and then to derive a feasible periodic schedule from the output of the program (and to describe this schedule in polynomial size too).…”

Non‐clairvoyant reduction algorithms for heterogeneous platforms

“…Reductions have been used in distributed programs for years, and standards such as Message Passing Interface (MPI) usually include a ‘reduce’ function together with other collective communications (see for experimental comparisons). Many algorithms have been introduced to optimize this operation on various platforms, with homogeneous or heterogeneous communication costs . Recently, this operation has received more attention because of the success of the MapReduce framework , which has been popularized by Google.…”