Load balancing and skew resilience for parallel joins

Abstract

Abstract-We address the problem of load balancing for parallel joins. We show that the distribution of input data received and the output data produced by worker machines are both important for performance. As a result, previous work, which optimizes either for input or output, stands ineffective for load balancing. To that end, we propose a multi-stage load-balancing algorithm which considers the properties of both input and output data through sampling of the original join matrix. To do this efficiently, we propose a novel category of equi-weight histograms. To build them, we exploit state-of-the-art computational geometry algorithms for rectangle tiling. To our knowledge, we are the first to employ tiling algorithms for join load-balancing. In addition, we propose a novel, join-specialized tiling algorithm that has drastically lower time and space complexity than existing algorithms. Experiments show that our scheme outperforms stateof-the-art techniques by up to a factor of 15.

D-JB: An Online Join Method for Skewed and Varied Data Streams

“…In data stream processing, DYNAMIC [4] supports adaptive repartitioning according to the change of data streams. To ensure the load balancing and skew resilience, Aleksandar el.al [6] proposed a multi-stage load-balancing algorithm by using a novel category of equi-weight histograms. However, [4,6] assumes that the number of partitions must be 2 n .…”

“…To ensure the load balancing and skew resilience, Aleksandar el.al [6] proposed a multi-stage load-balancing algorithm by using a novel category of equi-weight histograms. However, [4,6] assumes that the number of partitions must be 2 n . So, the matrix structure suffers from bad flexibility.…”

GPU processing of theta‐joins

“…Koumarelas et al deal with the issue of preprocessing the JM, when selectivity information is known a‐priori. Victorovic et al improves on the previous study for a specific form of JMs, termed as monotonic. Beame et al, Zhang et al, and Cu et al investigate the case where multiple relations are joined in a single step.…”

Submodularity of Distributed Join Computation

“…Input partitioning was also considered for the more general problem of distributed theta-join computation. Vitorovic et al [38] propose a new tiling algorithm to partition the join matrix in a balanced way, improving over earlier work by Okcan and Riedewald [27]. However, the authors themselves point out that for equi-joins one should instead rely on a specialized solution such as [2], because general theta-join approaches do not exploit the strong structural properties of key-equality based matching in equi-joins.…”