Skew strikes back

Abstract

Evaluating the relational join is one of the central algorithmic and most well-studied problems in database systems. A staggering number of variants have been considered including Block-Nested loop join, Hash-Join, Grace, Sort-merge (see Grafe [17] for a survey, and [4,7,24] for discussions of more modern issues). Commercial database engines use finely tuned join heuristics that take into account a wide variety of factors including the selectivity of various predicates, memory, IO, etc. This study of join queries notwithstanding, the textbook description of join processing is suboptimal. This survey describes recent results on join algorithms that have provable worst-case optimality runtime guarantees. We survey recent work and provide a simpler and unified description of these algorithms that we hope is useful for theory-minded readers, algorithm designers, and systems implementors.Much of this progress can be understood by thinking about a simple join evaluation problem that we illustrate with the so-called triangle query, a query that has become increasingly popular in the last decade with the advent of social networks, biological motifs, and graph databases [36,37] Suppose that one is given a graph with N edges, how many distinct triangles can there be in the graph?A first bound is to say that there are at most N edges, and hence at most OpN 3 q triangles. A bit more thought suggests that every triangle is indexed by any two of its sides and hence there at most OpN 2 q triangles. However, the correct, tight, and non-trivial asymptotic is OpN 3{2 q.

Optimal Join Algorithms Meet Top-k

“…A variety of WCO join algorithms have been proposed to match the AGM bound [54,72,77,78,91]. In contrast to the common "two-relations-at-a-time" approach, i.e., binary join plans, favored by database optimizers, they take a more "holistic" approach by computing a multiway join directly.…”

“…No matter the join order for a binary join plan, 5 the first binary join produces O(n 2 ) intermediate results, even though the AGM bound shows that final output size cannot exceed n 1.5 . As a consequence, the binary-join approach has complexitỹ O(n 2 ), while a WCO join algorithm like Generic-Join [78] or NPRR [77] computes the output in timeÕ(n 1.5 ).…”

“…This enables lower widths compared to decompositions to a single tree. For example, on the 4-cycle query both the WCO Generic-Join [78] and approaches based on single-tree decompositions have complexityÕ(n 2 ), the former due to worst-case output size being quadratic in input size and the latter due to the fractional hypertree width being d = 2. In contrast, submodular width is 1.5 and hence algorithms like PANDA [6] that rely on decompositions into multiple trees achieve complexityÕ(n 1.5 +r ), which is better for small output size r = O(n 1.5 ).…”

Recent Results on Cardinality Estimation and Information Theoretic Inequalities

“…Thus, while studied in a completely different context, Loomis-Whitney's inequality is our earliest known answer to determining the worst-case size bound of a special class of conjunctive queries. The class of queries coming from Loomis-Whitney setup are now referred to as Loomis-Whitney queries [27,28].…”

“…Hence, we will be almost always brief in presenting materials, favoring to convey the main ideas instead of being correct the complete to the last detail. The reader is encouraged to refer to the references (such as [1,14,26,28]) for more background and survey presentations.…”