Communication Steps for Parallel Query Processing

Abstract

We consider the problem of computing a relational query q on a large input database of size n, using a large number p of servers. The computation is performed in rounds, and each server can receive only O(n/p 1−ε ) bits of data, where ε ∈ [0, 1] is a parameter that controls replication. We examine how many global communication steps are needed to compute q. We establish both lower and upper bounds, in two settings. For a single round of communication, we give lower bounds in the strongest possible model, where arbitrary bits may be exchanged; we show that any algorithm requires ε ≥ 1 − 1/τ * , where τ * is the fractional vertex cover of the hypergraph of q. We also give an algorithm that matches the lower bound for a specific class of databases. For multiple rounds of communication, we present lower bounds in a model where routing decisions for a tuple are tuple-based. We show that for the class of tree-like queries there exists a tradeoff between the number of rounds and the space exponent ε. The lower bounds for multiple rounds are the first of their kind. Our results also imply that transitive closure cannot be computed in O(1) rounds of communication.than main memory access. In addition, any data reshuffling requires a global synchronization of all servers, which also comes at significant cost; for example, everyone needs to wait for the slowest server, and, worse, in the case of a straggler, or a local node failure, everyone must wait for the full recovery. Thus, the dominating complexity parameters in big data query processing are the number of communication steps, and the amount of data being exchanged.MapReduce-related models Several computation models have been proposed in order to understand the power of MapReduce and related massively parallel programming methods [9,16,17,1]. These all identify the number of communication steps/rounds as a main complexity parameter, but differ in their treatment of the communication.The first of these models was the MUD (Massive, Unordered, Distributed) model of Feldman et al. [9]. It takes as input a sequence of elements and applies a binary merge operation repeatedly, until obtaining a final result, similarly to a User Defined Aggregate in database systems. The paper compares MUD with streaming algorithms: a streaming algorithm can trivially simulate MUD, and the converse is also possible if the merge operators are computationally powerful (beyond PTIME).Karloff et al.[16] define MRC, a class of multi-round algorithms based on using the MapReduce primitive as the sole building block, and fixing specific parameters for balanced processing. The number of processors p is Θ(N 1− ), and each can exchange MapReduce outputs expressible in Θ(N 1− ) bits per step, resulting in Θ(N 2−2 ) total storage among the processors on a problem of size N. Their focus was algorithmic, showing simulations of other parallel models by MRC, as well as the power of two round algorithms for specific problems.Lower bounds for the single round MapReduce model are first discussed by Afrati et al....

From Theory to Practice

“…Afrati and Ullman model this as a non-convex optimization problem, which is difficult to solve. Beame et al [8,9] model it as a linear optimization problem, and establish a tight connection between the optimal shares and a fractional edge packing of the query hypergraph. This problem is easily solvable but leads to a fractional solution for the shares.…”

“…We build on two lines of work that introduce a novel parallel [5,8,9], and a novel sequential [23,33] algorithm respectively. While the former has been studied only theoretically, the latter is in use in the LogicBlox DBMS.…”

“…For parallel computation, Afrati and Ullman [5] have described an algorithm that computes any multi-join query in a single communication round. Beame at al [8,9] refined this algorithm, calling it HyperCube, and performed a theoretical analysis proving that it is optimal. However, the optimality criterion described by Beame et.…”

“…We then consider practical aspects of both algorithms. For the HyperCube algorithm, we design a new approach to optimize the number of server shares per variable, which always results in an integral number of shares, thus overcoming a key limitation of prior work [8,9]. We demonstrate empirically its performance; for example we show that it cuts the maximum amount of data per worker in half compared with a naïve application of existing theoretical methods (Figure 11), and that its workload per server is never larger than 1.06 times the theoretically optimal load.…”

Deciding Determinism with Fairness for Simple Transducer Networks

“…If the task at hand is not monotone, however, a form of coordination has to be used to enforce determinism [Zinn et al 2012;]. There is ongoing theoretical research about the cost and complexity of coordination [Koutris and Suciu 2011;Beame et al 2013].…”