A parallel hash join algorithm for managing data skew

Wolf, Joel L.; Yu, Philip S.; Turek, John; Dias, Daniel M.

doi:10.1109/71.250117

Cited by 40 publications

(16 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Wolf et al proposed heuristics for distributed sort-merge [16] and hash-join [17] algorithms to achieve load balancing in the presence of skew. However, their approach targets CPU bound systems and does not apply when network is the bottleneck.…”

Section: Related Workmentioning

confidence: 99%

Locality-sensitive operators for parallel main-memory database clusters

Rödiger

Mühlbauer

Unterbrunner³

et al. 2014

2014 IEEE 30th International Conference on Data Engineering

View full text Add to dashboard Cite

Abstract-The growth in compute speed has outpaced the growth in network bandwidth over the last decades. This has led to an increasing performance gap between local and distributed processing. A parallel database cluster thus has to maximize the locality of query processing. A common technique to this end is to co-partition relations to avoid expensive data shuffling across the network. However, this is limited to one attribute per relation and is expensive to maintain in the face of updates. Other attributes often exhibit a fuzzy co-location due to correlations with the distribution key but current approaches do not leverage this.In this paper, we introduce locality-sensitive data shuffling, which can dramatically reduce the amount of network communication for distributed operators such as join and aggregation. We present four novel techniques: (i) optimal partition assignment exploits locality to reduce the network phase duration; (ii) communication scheduling avoids bandwidth underutilization due to cross traffic; (iii) adaptive radix partitioning retains locality during data repartitioning and handles value skew gracefully; and (iv) selective broadcast reduces network communication in the presence of extreme value skew or large numbers of duplicates. We present comprehensive experimental results, which show that our techniques can improve performance by up to factor of 5 for fuzzy co-location and a factor of 3 for inputs with value skew.

show abstract

Section: Related Workmentioning

confidence: 99%

Locality-sensitive operators for parallel main-memory database clusters

Rödiger

Mühlbauer

Unterbrunner³

et al. 2014

2014 IEEE 30th International Conference on Data Engineering

View full text Add to dashboard Cite

show abstract

“…Malleable tasks are allowed to be preempted and change the number of cores during execution. Scheduling malleable tasks is a promising technique for gaining computational speedup when solving large scheduling problems on parallel and distributed computers [8], [34]. Real applications for malleable tasks have been presented among others in [2] for simulating molecular dynamics, in [12] for Cholesky factorization, in [4] for operational oceanography and in [5] for berth and quay allocation.…”

Section: Related Workmentioning

confidence: 99%

Task Scheduling on Adaptive Multi-Core

Pricopi

Mitra

2014

IEEE Trans. Comput.

View full text Add to dashboard Cite

Abstract-Multi-cores have become ubiquitous both in the general-purpose computing and the embedded domain. The current technology trends show that the number of on-chip cores is rapidly increasing, while their complexity is decreasing due to power and thermal constraints. Increasing number of simple cores enable parallel applications benefit from abundant threadlevel parallelism (TLP), while sequential fragments suffer from poor exploitation of instruction-level parallelism (ILP). Recent research has proposed adaptive multi-core architectures that are capable of coalescing simple physical cores to create more complex virtual cores so as to accelerate sequential code. Such adaptive architectures can seamlessly exploit both ILP and TLP.The goal of this paper is to quantitatively characterize the performance potential of adaptive multi-core architectures. Previous research have primarily focused on only sequential workload on adaptive multi-cores. We address a more realistic scenario where parallel and sequential applications co-exist on an adaptive multi-core platform. Scheduling tasks on adaptive architectures reveals challenging resource allocation problems for the existing schedulers. We construct offline and online schedulers that intelligently reconfigure and allocate the cores to the applications so as to minimize the overall makespan under the constraints of a realistic adaptive multi-core architecture. Experimental results reveal that adaptive multi-core architectures can substantially decrease the makespan compared to both static symmetric and asymmetric multi-core architectures.

show abstract

“…In the presence of data skew, we have to modify the corresponding formula accordingly [ 111. Studies on the effect of data skew can be found in [17], [18], [42]. For ease of exposing the concept of segmented right-deep trees, we assume the aggregate memory in the system can accommodate a few entire relations for pipelining.…”

Section: Preliminariesmentioning

confidence: 99%

Applying segmented right-deep trees to pipelining multiple hash joins

Chen

Lo²,

Yu³

et al. 1995

IEEE Trans. Knowl. Data Eng.

Self Cite

View full text Add to dashboard Cite

Abstract-The pipelined execution of multijoin queries in a multiprocessor-based database system is explored in this paper. Using hash-based joins, multiple joins can be pipelined so that the early results from a join, before the whole join is completed, are sent to the next join for processing. The execution of a query is usually denoted by a query execution tree. To improve the execution of pipelined hash joins, an innovative approach on query execution tree selection is proposed to exploit segmented rightdeep trees, which are bushy trees of right-deep subtrees. We first derive an analytical mode1 for the execution of a pipeline segment, and then, in light of the model, develop heuristic schemes to determine the query execution plan based on a segmented rightdeep tree so that the query can be efficiently executed. As shown by our simulation, the proposed approach, without incurring additional overhead on plan execution, possesses more flexibility in query plan generation, and can lead to query plans of better performance than those achievable by the previous schemes using right-deep trees.

show abstract

A parallel hash join algorithm for managing data skew

Cited by 40 publications

References 25 publications

Locality-sensitive operators for parallel main-memory database clusters

Locality-sensitive operators for parallel main-memory database clusters

Task Scheduling on Adaptive Multi-Core

Applying segmented right-deep trees to pipelining multiple hash joins

Contact Info

Product

Resources

About