Communication steps for parallel query processing

Abstract

We consider the problem of computing a relational query q on a large input database of size n, using a large number p of servers. The computation is performed in rounds, and each server can receive only O(n/p 1−ε ) bits of data, where ε ∈ [0, 1] is a parameter that controls replication. We examine how many global communication steps are needed to compute q. We establish both lower and upper bounds, in two settings. For a single round of communication, we give lower bounds in the strongest possible model, where arbitrary bits may be exchanged; we show that any algorithm requires ε ≥ 1−1/τ * , where τ * is the fractional vertex cover of the hypergraph of q. We also give an algorithm that matches the lower bound for a specific class of databases. For multiple rounds of communication, we present lower bounds in a model where routing decisions for a tuple are tuple-based. We show that for the class of tree-like queries there exists a tradeoff between the number of rounds and the space exponent ε. The lower bounds for multiple rounds are the first of their kind. Our results also imply that transitive closure cannot be computed in O(1) rounds of communication.

On the Computational Complexity of MapReduce

“…We reformulate the definition of [12] to make a uniform model and to more finely track the parameters involved (Section 3.2). In addition, we point out that our results hold for other important models of parallel computations, including Valiant's Bulk-Synchronous Processing (BSP) model [20] and the Massively Parallel Communication (MPC) model of Beame et al [2]. (Section 3.3).…”

“…There are a number of MapReduce-like models in the literature, including the MRC model of Karloff et al [12], the "mud" algorithms of Feldman et al [6], Valiant's BSP model [20], the MPC model of Beame et al [2], and extensions or generalizations of these, e.g. [8].…”

“…al. [2]. The main difference between BSP and MapReduce is that in the BSP models the key-value pairs and the shuffling steps needed to redistribute them are replaced with point-to-point messages.…”

“…al. 's paper [2] does not discuss the uniformity of the model. The main distinguishing charateristic of the MPC model is that it introduces the number of processors p as an explicit paramter.…”

Big Data Begets Big Database Theory

“…Instead, we consider the following simple model, called the Massively Parallel Communication (MPC) model, introduced in [6]. There are a fixed number of servers, p, and the input data of size n is initially uniformly distributed on the servers; thus, each server holds O(n/p) data items.…”