Instance and Output Optimal Parallel Algorithms for Acyclic Joins

Abstract

Massively parallel join algorithms have received much attention in recent years, while most prior work has focused on worst-optimal algorithms. However, the worst-case optimality of these join algorithms relies on hard instances having very large output sizes, which rarely appear in practice. A stronger notion of optimality is output-optimal, which requires an algorithm to be optimal within the class of all instances sharing the same input and output size. An even stronger optimality is instance-optimal, i.e., the algorithm is optimal on every single instance, but this may not always be achievable.In the traditional RAM model of computation, the classical Yannakakis algorithm is instanceoptimal on any acyclic join. But in the massively parallel computation (MPC) model, the situation becomes much more complicated. We first show that for the class of r-hierarchical joins, instance-optimality can still be achieved in the MPC model. Then, we give a new MPC algorithm for an arbitrary acyclic join with load O ( IN p + √ IN·OUT p ), where IN, OUT are the input and output sizes of the join, and p is the number of servers in the MPC model. This improves the MPC version of the Yannakakis algorithm by an O( OUT IN ) factor. Furthermore, we show that this is output-optimal when OUT = O(p · IN), for every acyclic but non-r-hierarchical join. Finally, we give the first output-sensitive lower bound for the triangle join in the MPC model, showing that it is inherently more difficult than acyclic joins. other servers, receives messages from other servers, and then does some local computation. The complexity of the algorithm is measured by the number of rounds and the load, denoted as L, which is the maximum message size received by any server in any round., all problems can be solved trivially in one round by simply sending all data to one server. Initial efforts were mostly spent on what can be done in a single round of computation [3,26,7,8,24,26], but recently, more interest has been given to multi-round (but still a constant) algorithms [2,22,24], since new main memory based systems, such as Spark and Flink, have much lower overhead per round than previous generations like Hadoop.The MPC model can be considered as a simplified version of the BSP model [32], but it has enjoyed more popularity in recent years. This is mostly because the BSP model takes too many measures into consideration, such as communication costs, local computation time, memory consumption, etc. The MPC model unifies all these costs with one parameter L, which makes the model much simpler. Meanwhile, although L is defined as the maximum incoming message size of a server, it is also closely related with the local computation time and memory consumption, which are both increasing functions of L. Thus, L serves as a good surrogate of these other cost measures. This is also why the MPC model does not limit the outgoing message size of a server, which is less relevant to other costs.All our algorithms work under the mild assumption IN ≥ p 1+ where > 0 is any sma...

Parallel Acyclic Joins with Canonical Edge Covers

“…The fourth class comprises all the acyclic queries, which were recently solved by Hu [8] optimally. It is worth pointing out that Hu's algorithm subsumes an earlier algorithm of [9] which is worst-case optimal on a subclass of acyclic queries.…”

Vertex-centric Parallel Computation of SQL Queries

“…In our vertex-centric join, the graph representation of the input database is never reshuffled, regardless of the query. The generalization of parallel sort-join algorithm to any acyclic join is presented in [33], and has a total communication cost of 𝑂 (𝐼 𝑁 + √ 𝐼 𝑁 • 𝑂𝑈𝑇 ). This outperforms our algorithm and parallel hash-join when the join output blows up to be lager than the input, but it is worse for selective joins and it is equivalent when the query involves only PK-FK joins.…”

“…On the theoretical front, we show that TAG-join's communication and computation complexities are competitive with those of the best-known parallel join algorithms [10,15,16,32,33,35] while avoiding the relation reshuffling these algorithms require Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored.…”

“…On the theoretical front, we show that TAG-join's communication and computation complexities are competitive with those of the best-known parallel join algorithms [13,20,21,39,40,42] while avoiding the relation reshuffling these algorithms require (for re-sorting or re-hashing) between individual join operations. TAG-join adapts techniques from the best sequential join algorithms (based on worst-case optimal bounds [51,52,58] and on generalized hypertree decompositions [34,35]), matching their computation complexity as well.…”

On Quasi Cycles in Hypergraph Databases

“…A good database scheme design would allow information to be retrieved easily and efficiently [10]. Acyclic databases are preferred due to the variety of desired computational properties it enjoys such as making query optimization easier than in the case of cyclic database and might be recognized in linear time [11][12][13][14][15][16][17].…”