Parallel sorting on a shared-nothing architecture using probabilistic splitting

Abstract

We consider the problem of external sorting in a shared-nothing multiprocessor. A critical step in the algorithms we consider is to determine the range of sort keys to be handled by e a c h processor. We consider two t e c hniques for determining these ranges of sort keys: exact splitting, using a parallel version of the algorithm proposed by I y er, Ricard, and Varman and probabilistic splitting, which uses sampling to estimate quantiles. We present analytic results showing that probabilistic splitting performs better than exact splitting. Finally, w e present experimental results from an implementation of sorting via probabilistic splitting in the Gamma parallel database machine.

Distributed join algorithms on thousands of cores

“…Merge-based sorting algorithms combine data from two or more processes [7]. Splitter-based approaches try to subdivide the input into chunks of roughly equal size [13,16,22,25,36]. The latter category utilize minimal data movement because the data only moves during the split operation.…”

Alphasort: A cache-sensitive parallel external sort

“…DeWitt, Naughton, and Schneider's efforts on an Intel Hypercube was the fastest reported time: 58.3 seconds using 32 processors, 32 disks and 224 MB of memory [9]. Baugsto, Greispland and Kamberbeek mentioned a 40-second sort on a 100-processor 100-disk system [4].…”

“…Despite this risk, QuickSort is widely used because, in practice, it has superior performance. Baugsto, Bitton, Beck, Graefe, and DeWitt used QuickSort [4,6,7,9,11]. On the other hand, Tsukerman and Weinberger used replacementselection [21,22].…”

Flux: an adaptive partitioning operator for continuous query systems

“…Early work concentrated on parallelizing individual, traditional content-sensitive operators like hybrid-hash join [25] and sort (e.g., [11,20,1]). The abstractions which inspired Flux, Exchange [12] and RiverDQ [23], were proposed to compose such operators into a dataflow.…”

“…The abstractions which inspired Flux, Exchange [12] and RiverDQ [23], were proposed to compose such operators into a dataflow. In [10] and [9], the authors present practical techniques for handling data skew for a hash join and external sort, respectively. These techniques rely on sampling a static data set, which is infeasible in the streaming scenario.…”