Implementing Parallel Google Map-Reduce in Eden

Abstract

Recent publications have emphasised map-reduce as a general programming model (labelled Google map-reduce), and described existing high-performance implementations for large data sets. We present two parallel implementations for this Google map-reduce skeleton, one following earlier work, and one optimised version, in the parallel Haskell extension Eden. Eden's specific features, like lazy stream processing, dynamic reply channels, and nondeterministic stream merging, support the efficient implementation of the complex coordination structure of this skeleton. We compare the two implementations of the Google map-reduce skeleton in usage and performance, and deliver runtime analyses for example applications. Although very flexible, the Google mapreduce skeleton is often too general, and typical examples reveal a better runtime behaviour using alternative skeletons.

Skeleton Composition Using Remote Data

“…Using these skeletons with their remote data interfaces enables us to define a sequence consisting of a parallel map, a parallel transpose (realised using the all-to-all skeleton) and a second parallel map. This can be useful in an implementation of a parallel FFT skeleton [8] or a Google Map-Reduce skeleton [4]. In [4,8], corresponding parallel map-transpose skeletons have been defined as monolithic skeletons without composing simpler skeletons.…”

“…This can be useful in an implementation of a parallel FFT skeleton [8] or a Google Map-Reduce skeleton [4]. In [4,8], corresponding parallel map-transpose skeletons have been defined as monolithic skeletons without composing simpler skeletons. With the remote data interface, we can define the same skeleton as a composition of the three component skeletons.…”

Monitoring and Notification System Based on Risk Analysis Using Map-Reduce Framework

“…The MapReduce programming model is designed to process large volumes of data in parallel by dividing a job into a set of independent tasks [8]- [10]. The job is referred to here as a full MapReduce program, which is the execution of a Mapper or Reducer across a set of data.…”

Eden – Parallel Functional Programming with Haskell

“…It splits the input list into two halves, keeps the first half for local evaluation and creates a child process on PE 2 for sorting the second half. The remaining ticket list [3..noPe] is unshuffled into the two lists [3,5,7] and [4,6,8]. The first sublist is kept locally while the child process gets the second one.…”

“…Several parallel map implementations have been discussed and analysed in [33]. An Eden implementation of the large-scale map-and-reduce programming model proposed by Google [18] has been investigated in [6,4]. Hierarchical master-worker schemes with several layers of masters and submasters have been presented in [7].…”