Asynchronous parallel disk sorting

Abstract

We develop an algorithm for parallel disk sorting, whose I/O cost approaches the lower bound and that guarantees almost perfect overlap between I/O and computation. Previous algorithms have either suboptimal I/O volume or cannot guarantee that I/O and computations can always be overlapped. We give an efficient implementation that can (at least) compete with the best practical implementations but gives additional performance guarantees. For the experiments we have configured a state of the art machine that can sustain full bandwidth I/O with eight disks and is very cost effective.

A Simple Optimal Randomized Algorithm for Sorting on the PDM

“…The second kind of sorting algorithms on the PDM are based on R-way merging for some suitable value of R that minimizes the number of passes through the data for the given size of internal memory [1,6,11,16,18].…”

“…The RC scheduling resulted in optimal distributed sort (RCD) and optimal mergesort (RCM) via duality [13]. These have been shown to be very practical [11]. However, these algorithms have been shown to be optimal only in expectation and no high probability bounds have been derived.…”

Better external memory suffix array construction

“…Our experimental platform has two 2.0 GHz Intel Xeon processors, one GByte of RAM, and we use four 80 GByte IBM 120GXP disks. Refer to [11] for a performance evaluation of this machine whose cost was 2500 Euro in July 2002. The following instances have been considered: Random2: Two concatenated copies of a Random string of length n/2.…”

Stxxl: Standard Template Library for XXL Data Sets

“…Both sorters are highly efficient parallel disk implementations. The algorithm they implement guarantees close to optimal I/O volume and almost perfect overlapping between I/O and computation [16]. The performance of the sorters scales well.…”

“…The input of the sorter may be an object complying to Stxxl stream interface. As the STL-user layer sorter, the pipelined sorter is an implementation of parallel disk merge sort [16] that overlaps I/O and computation. The implementation of stream::sort relies on two classes that encapsulate the two phases of the algorithm: sorted run formation (class runs creator) and run merging (runs merger).…”