Scalable communication protocols for dynamic sparse data exchange

Abstract

Many large-scale parallel programs follow a bulk synchronous parallel (BSP) structure with distinct computation and communication phases. Although the communication phase in such programs may involve all (or large numbers) of the participating processes, the actual communication operations are usually sparse in nature. As a result, communication phases are typically expressed explicitly using point-to-point communication operations or collective operations. We define the dynamic sparse data-exchange (DSDE) problem and derive bounds in the well known LogGP model. While current approaches work well with static applications, they run into limitations as modern applications grow in scale, and as the problems that are being solved become increasingly irregular and dynamic. To enable the compact and efficient expression of the communication phase, we develop suitable sparse communication protocols for irregular applications at large scale. We discuss different irregular applications and show the sparsity in the communication for real-world input data. We discuss the time and memory complexity of commonly used protocols for the DSDE problem and develop N BX -a novel fast algorithm with constant memory overhead for solving it. Algorithm N BX improves the runtime of a sparse dataexchange among 8,192 processors on BlueGene/P by a factor of 5.6. In an application study, we show improvements of up to a factor of 28.9 for a parallel breadth first search on 8,192 BlueGene/P processors.

A sliding characteristic interface condition for direct numerical simulations

“…A solution to this problem is to perform step 6 using a variant of the nonblocking consensus algorithm described by Höfler et al [9]. All sends and receives are tested for completion, a nonblocking barrier posted once the tests are passed, and incoming messages probed for with MPI Iprobe, all in a loop which only exits if the nonblocking barrier is reached by all participating processes.…”

“…This variant differs slightly from Höfler's -tests for completion take place before probing. This is because the probe (and subsequent communication) is usually unnecessary, and both MPI Iprobe and MPI Ssend do have overhead (see [9] for some measurements pertaining to MPI Ssend). , is zero in either reference frame (this would not be true if differentiating across the interface, but it is so for a change of (inertial) reference frame), and…”

Adoption protocols for fanout-optimal fault-tolerant termination detection

“…For example, in preparing the data exchange necessary to obtain column data in sparse matrix-vector multiplication, this would be the number of processors having entries on that column [7]. We use the fanout, denoted f , to describe the bounds on our algorithms' costs.…”

“…At the same time, the popularity of distributed parallel programming systems that implement high degrees of dynamic behavior, such as asynchronous tasks [1], work stealing [4,11,12], and messagedriven execution [8,18,19], are increasing. Unlike in bulk synchronous parallel programs, and even in dynamic data exchanges within BSP programs [7], there is often no clear global indication of when some particular distributed computation is complete. Thus, they instead rely on termination detection algorithms to provide that indication.…”

Cosmic microwave background map-making at the petascale and beyond

“…Hoefler et al [14] study this problem and its variations, and design new, specialized collectives to address this problem. Hoefler and Träff [15] also make the case for better support of "sparse" communication patterns within MPI, where the sparsity refers to the number of communicating processes.…”