A Spinning Join That Does Not Get Dizzy

Frey, Philip W.; Gonçalves, Romulo; Kersten, Martin; Teubner, Jens

doi:10.1109/icdcs.2010.23

Cited by 23 publications

(24 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Goncalves et al [16,15] and Frey et al [12,13] have developed a join algorithm, called cyclo-join, suited for ring network topology networks. In the setup phase of the cyclojoin, both relations are fragmented and distributed over all n machines.…”

Section: Distributed Join Algorithmsmentioning

confidence: 99%

Rack-Scale In-Memory Join Processing using RDMA

Barthels

Loesing

Alonso

et al. 2015

Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

Database systems running on a cluster of machines, i.e. rack-scale databases, are a common architecture for many large databases and data appliances. As the data movement across machines is often a significant bottleneck, these systems typically use a low-latency, high-throughput network such as InfiniBand. To achieve the necessary performance, parallel join algorithms must take advantage of the primitives provided by the network to speed up data transfer.In this paper we focus on implementing parallel in-memory joins using Remote Direct Memory Access (RDMA), a communication mechanism to transfer data directly into the memory of a remote machine. The results of this paper are, to our knowledge, the first detailed analysis of parallel hash joins using RDMA. To capture their behavior independently of the network characteristics, we develop an analytical model and test our implementation on two different types of networks. The experimental results show that the model is accurate and the resulting distributed join exhibits good performance.

show abstract

Section: Distributed Join Algorithmsmentioning

confidence: 99%

Rack-Scale In-Memory Join Processing using RDMA

Barthels

Loesing

Alonso

et al. 2015

Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data

View full text Add to dashboard Cite

show abstract

“…Frey et al [18] proposed the idea of the data-cyclotron, a system with a ring-shaped network topology in which fragments of one relation are constantly transmitted from one machine to the next. Barthels et al [6] proposed to use RDMA in order to reduce the communication costs of large data transfers and hide the remote memory access latencies by interleaving communication and computation.…”

Section: Related Workmentioning

confidence: 99%

Distributed join algorithms on thousands of cores

et al. 2017

View full text Add to dashboard Cite

Traditional database operators such as joins are relevant not only in the context of database engines but also as a building block in many computational and machine learning algorithms. With the advent of big data, there is an increasing demand for efficient join algorithms that can scale with the input data size and the available hardware resources.In this paper, we explore the implementation of distributed join algorithms in systems with several thousand cores connected by a low-latency network as used in high performance computing systems or data centers. We compare radix hash join to sort-merge join algorithms and discuss their implementation at this scale. In the paper, we explain how to use MPI to implement joins, show the impact and advantages of RDMA, discuss the importance of network scheduling, and study the relative performance of sorting vs. hashing. The experimental results show that the algorithms we present scale well with the number of cores, reaching a throughput of 48.7 billion input tuples per second on 4,096 cores.

show abstract

“…Algorithm: 2 Block Nested Loop Join (Block-at-a-time) [9] for each block bR of 'r' do {blocks of relation 'r' are scanned one by one.…”

Section: Block Nested Loop Join (Bnlj-one Block At a Time)mentioning

confidence: 99%

“…The block nested loop joins algorithm works by reading a block of tuples, from the outer and inner relation [10]. In BNLJ (one block at a time) chunks of each relation is transferred from hard disk to main memory where join operations is performed [9].…”

Section: Block Nested Loop Join (Bnlj-one Block At a Time)mentioning

confidence: 99%

An Efficient Approach of Block Nested Loop Algorithm Based on Rate of Block Transfer

Shukla¹,

Arora²,

Pandey³

et al. 2011

IJCA

View full text Add to dashboard Cite

In this paper, an approach has been proposed that makes the processing of join operation in database systems more efficient. In join operation processing relations that take part in the join process are required to be transferred to the main memory (RAM) from hard disk. In join operation processing when block nested loop algorithm is used to perform join between relations and multiple blocks of the relations that take part in the joining process are transferred from hard disk to main memory than in this case the main memory buffer allotted to the blocks of relation. Using this approach, multiple blocks are transferred for the relations that participates in the join operation processing, instead of transferring blocks one by one for each relation (or multiple blocks for one relation) without worrying about the large and small databases size. When this new approach is applied, the rate of block transfer during join operation processing using block nested loop algorithm get minimizes and join query processing become efficient, without loosing the level of complexity of the previous algorithms of block nested loop join(BNLJ). General TermsAlgorithm for join operation processing based on rate of block transfer.

show abstract

A Spinning Join That Does Not Get Dizzy

Cited by 23 publications

References 18 publications

Rack-Scale In-Memory Join Processing using RDMA

Rack-Scale In-Memory Join Processing using RDMA

Distributed join algorithms on thousands of cores

An Efficient Approach of Block Nested Loop Algorithm Based on Rate of Block Transfer

Contact Info

Product

Resources

About