Logarithmic time cost optimal parallel sorting is not yet fast in practice

Abstract

When looking for new and faster parallel sorting algorithms for use in massively parallel systems it is tempting to investigate promising alternatives from the large body of research d o n e o n parallel sorting in the eld of theoretical computer science. Such \theoretical" algorithms are mainly described for the PRAM (Parallel Random Access Machine) model of computation 13,26]. This paper shows how this kind of investigation can be done on a simple but versatile environment f o r programming and measuring of PRAM algorithms 18, 1 9 ]. The practical value of Cole's Parallel Merge Sort algorithm 10,11] have b e e n i n vestigated by comparing it with Batcher's bitonic sorting 5]. The O(logn) time consumption of Cole's algorithm implies that it must be faster than bitonic sorting which is O(log 2 n) time|if n is large enough. However, we h a ve found that bitonic sorting is faster as long as n is less than 1:2 10 21 , i.e. more than 1 Giga Tera items!. Consequently, Cole's logarithmic time algorithm is not fast in practice. I n troduction and MotivationThe work reported in this paper is an attempt to lessen the gap between theory and practice within the eld of parallel computing. Within theoretical computer science, parallel algorithms are mainly compared by using asymptotical analysis (O-notation). This paper gives an example on how the analysis of implemented algorithms on nite problems provides new and more practically oriented results than those traditionally obtained by asymptotical analysis. Parallel Complexity Theory|A Rich Source for Parallel AlgorithmsIn the past years there has been an increased interest in parallel algorithms. In the eld of parallel complexity theory the so called Nick's Class (NC) has been given a lot of attention. Problems belonging to this complexity class may b e solved by algorithms with polylogarithmic (i.e. of O(log k (n)) where k is a constant a n d n is the problem size) running time and a polynomial requirement for processors. A lot of parallel algorithms have recently been presented to prove membership in this class for various problems. This kind of algorithms are frequently denoted NC-algorithms. Although the main motivation for these new algorithms often is to show that a g i v en problem is in NC, the algorithms may also be taken as proposals for parallel algorithms that are \fast in practice". The Gap Between Theory and PracticeUnfortunately, i t m a y b e v ery hard to assess the practical value of NC-algorithms.Asymptotical analysis Order{notation is a very useful tool when the aim is to prove membership of complexity classes or if asymptotic behavior for some other reason is \detailed enough". It makes it possible to describe and analyze algorithms at a very high level. However, it also makes it possible to hide (willingly or unwillingly) a lot of details which cannot be omitted when algorithms are compared for \re-alistic" problems of large but limited size. 1Unrealistic machine model Few of these theoretical algorithms are described as implementations on rea...

Deterministic Parallel Sorting Algorithm for 2-D Mesh of Connected Computers

“…Several optimal sequential algorithms are known, such as Quicksort [4] and Heapsort [4] . A vast number of parallel sorting algorithms have been described in the literatures, some of them are cost-optimal PRAM parallel sorting algorithms [16] . The most fundamental one is called Cole's Merge Sort [5,14] , it sorts n numbers using n processors in time O (logn).…”

A General-Purpose Parallel Sorting Algorithm

“…See, for example, 1,2,12]. Most of these papers have dealt with the problem from a theoretical point of view, neglecting many issues that are important in a practical implementation of a parallel sorting algorithm 4, 10,14]. This paper describes a practical parallel sorting algorithm which is suitable for e cient general-purpose internal sorting.…”

Efficient parallel algorithms for closest point problems.

“…Moreover, few implementations have delivered reasonable performance. This is primarily due to the use of unrealistic computational models and high constant factors hidden by asymptotic analysis [Nat90].…”

“…One canonical example of this is Cole's pipelined, parallel mergesort [ACG89]. While it is a tour-de-force of elegant algorithm design and analysis, and contains many beautiful ideas, the algorithm is so complicated and uses so much global data movement that it simply can't compete against simpler algorithms such as radix sort [Ble89,Nat90].…”