Modeling Parallel Bandwidth: Local versus Global Restrictions

Abstract

Abstract.Recently there has been an increasing interest in models of parallel computation that account for the bandwidth limitations in communication networks. Some models (e.g., BSP, LOGP, and QSM) account for bandwidth limitations using a per-processor parameter g > 1, such that each processor can send/receive at most h messages in g · h time. Other models (e.g., PRAM(m)) account for bandwidth limitations as an aggregate parameter m < p, such that the p processors can send at most m messages in total at each step.This paper provides the first detailed study of the algorithmic implications of modeling parallel bandwidth as a per-processor (local) limitation versus an aggregate (global) limitation. We consider a number of basic problems such as broadcasting, parity, summation, and sorting, and give several new upper and lower time bounds that demonstrate the advantage of globally limited models over locally limited models given the same aggregate bandwidth (i.e., p·1/g = m). In general, globally limited models have a possible advantage whenever there is an imbalance in the number of messages sent/received by the processors. To exploit this advantage, the processors must schedule the sending of messages so as to respect the aggregate bandwidth limit. We present a new parallel scheduling algorithm for globally limited models that enable an unknown, arbitrarily unbalanced set of messages to be sent through the limited bandwidth within a (1 + ε) factor of the optimal off-line schedule with high probability, even if the penalty for overloading the network is an exponential function of the overload. We also present a near-optimal algorithm for the case where long messages must be sent as flits in consecutive time steps, as well as for the case where new messages to be sent arrive dynamically over an infinite time line. These results consider both message passing (distributed memory) and shared memory scenarios, and improve upon the best results for the locally limited model by a factor of (g). Finally, we present results quantifying the power of concurrent reads in a globally limited bandwidth setting, including showing an ( p lg m/m lg p) time separation between the exclusive-read and the concurrent-read PRAM(m) models, which, when m p, greatly improves upon the 2 ( √ lg p) separation known previously.

Can a Shared-Memory Model Serve as a Bridging Model for Parallel Computation?

“…Such validation may reveal the primary importance of features not present in either the QSM, BSP, or LogP. For example, each of these models defines a single bandwidth parameter that reflects a per-processor bandwidth limitation; other recent work has considered variants of these models with an aggregate bandwidth limitation [1] or a hierarchical bandwidth limitation that accounts for network proximity [52], [25], [26], [46], [73]. Per-processor bandwidth limitations better model machines in which each processor has access to its "share" of the network bandwidth and no more, as well as machines for which the primary network bottleneck, in the absence of hot-spots, is in the processor-network interface.…”

“…A lower bound of (g lg n/lg g) for broadcasting to n processors is given in [1]; in contrast to an earlier lower bound for this problem on the BSP given in [45] this lower bound holds even if processors can acquire knowledge through nonreceipt of messages (i.e., by reading memory locations that were not updated by a recent write operation). We note that the same lower bound on time holds for the problem of broadcasting to n memory locations since any algorithm that broadcasts to n memory locations can broadcast to n processors in additional g units of time.…”

“…Then the time cost for the phase is max m op , g · m r w , κ . 1 The time of a QSM algorithm is the sum of the time costs for its phases. The work of a QSM algorithm is its processor-time product.…”

“…The following lower bounds for the QSM are given in [1]. The CRCW PRAM lower bound result of Beame and Håstad [13] gives a lower bound for the n-element parity, summation, list ranking, and sorting problems of (g · lg n/lg lg n) time on the QSM for either deterministic or randomized algorithms when the number of processors is polynomial in n, the size of the input.…”

Parallel Bridging Models and Their Impact on Algorithm Design

“…To overcome these problems, several proposals for so-called parallel bridging models have been developed: for example, the BSP model [16], the LogP model [6], the CGM model [7], and the QSM model [1].…”