Sort vs. Hash revisited

Kim, Changkyu; Kaldewey, Tim; Lee, Victor W.; Sedlar, Eric; Nguyen, Anthony D.; Satish, Nadathur; Chhugani, Jatin; Blas, Andrea Di; Dubey, Pradeep

doi:10.14778/1687553.1687564

Cited by 221 publications

(47 citation statements)

References 31 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…In order to make our results comparable to previous work on join algorithms, we use the same workloads as the authors of [3,5,6,9,27]. The experiments focus on large-to-large joins with highly distinct key values.…”

Section: Methodsmentioning

confidence: 99%

“…As previous work [27,9,4,5,3], we investigate the join operation in isolation and do not materialize the output, i.e., we do not fetch additional data over the network after the join result has been computed. …”

Section: Input Selectivitymentioning

confidence: 99%

“…Comparing sort-merge and hash join algorithms has been the topic of recent work for joins on multi-core systems and efficient algorithms for both strategies have been proposed [1,3,4,5,9,27,30]. Kim et al [27] concluded that although modern hardware currently favors hash join algorithms, future processors with wider Single-InstructionMultiple-Data (SIMD) instructions would significantly speed up sort-merge joins.…”

Section: Related Workmentioning

confidence: 99%

“…Kim et al [27] concluded that although modern hardware currently favors hash join algorithms, future processors with wider Single-InstructionMultiple-Data (SIMD) instructions would significantly speed up sort-merge joins. Albutiu et al [1] presented a sort-merge join optimized for multi-socket NUMA machines.…”

Section: Related Workmentioning

confidence: 99%

“…Recent work on hash and sort-merge join algorithms for multi-core machines [1,3,5,9,27] and rack-scale data processing systems [6,33] has shown that carefully tuned distributed join implementations exhibit good performance. These algorithms have been designed for and evaluated on rack-scale systems with hundreds of CPU cores and limited inter-node network bandwidth.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Distributed join algorithms on thousands of cores

et al. 2017

View full text Add to dashboard Cite

Traditional database operators such as joins are relevant not only in the context of database engines but also as a building block in many computational and machine learning algorithms. With the advent of big data, there is an increasing demand for efficient join algorithms that can scale with the input data size and the available hardware resources.In this paper, we explore the implementation of distributed join algorithms in systems with several thousand cores connected by a low-latency network as used in high performance computing systems or data centers. We compare radix hash join to sort-merge join algorithms and discuss their implementation at this scale. In the paper, we explain how to use MPI to implement joins, show the impact and advantages of RDMA, discuss the importance of network scheduling, and study the relative performance of sorting vs. hashing. The experimental results show that the algorithms we present scale well with the number of cores, reaching a throughput of 48.7 billion input tuples per second on 4,096 cores.

show abstract

Section: Methodsmentioning

confidence: 99%

Section: Input Selectivitymentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Distributed join algorithms on thousands of cores

et al. 2017

View full text Add to dashboard Cite

show abstract

Nimble join: A parallel star join for main memory column‐stores

Sangat

Taniar

Indrawan-Santiago

et al. 2019

Concurrency and Computation

View full text Add to dashboard Cite

Summary Column‐stores perform significantly better than row‐stores on analytical workloads such as those found in data warehouses, decision support, and business intelligence applications. As mainstream data warehouses are growing into multi‐terabyte range, decision support queries should be processed in parallel to achieve adequate performance. Researchers of the column‐oriented join queries assume an unlimited reserve of main memory and focus on minimising execution time. However, some analytics require a large amount of memory to calculate intermediate results, and some interactive analytics require a fast initial response time even though queries need to process a large amount of data. Motivated by these requirements, we present a new progressive parallel star algorithm for main memory column‐stores known as “Nimble Join.” Equipped with multi‐attribute array table and a novel progressive materialisation technique, Nimble Join requires half the memory space and has two times faster initial response whilst having comparable execution time to the existing algorithm.

show abstract

Utilizing the column imprints to accelerate no‐partitioning hash joins in large‐scale edge systems

Liu

2020

Trans Emerging Tel Tech

View full text Add to dashboard Cite

With the increasing number of edge devices in large-scale edge systems, more and more data are collected to be processed. In such big data scenarios, there is a resurgence of interest in main-memory analytic databases because of the large RAM capacity of modern servers and the increasing demand for real-time analytic platforms. In such databases, join is at the heart of almost every query plan. Join also stays as a time-consuming operation when the denormalization overhead is too large to be applicable. However, the current implementations of these operations have not fully leveraged the new features (eg, SIMD, multi-core) provided by the modern hardware. The goal of this article is to design efficient algorithms for joins by judiciously exploiting every bit of RAM and all the available parallelisms in each processing unit. For join operations, hash joins have been studied, improved, and reexamined over decades.In this article, we propose to utilize a secondary index to improve hash joins without the physical partitioning. Specifically, in the build phase, the hash values are scattered evenly into the logical partitions of the hash table; in the probe phase, the secondary index is used as the hints to re-order the probing sequence, such that the locality of the hash probing is increased. We benchmark the performance of the proposed techniques in our column-store research prototype. Extensive experiments on the synthetic data and the real data show that our methods offer significant performance improvement over their counterparts.

show abstract

Sort vs. Hash revisited

Cited by 221 publications

References 31 publications

Distributed join algorithms on thousands of cores

Distributed join algorithms on thousands of cores

Nimble join: A parallel star join for main memory column‐stores

Utilizing the column imprints to accelerate no‐partitioning hash joins in large‐scale edge systems

Contact Info

Product

Resources

About