Deterministic parallel random-number generation for dynamic-multithreading platforms

Abstract

Existing concurrency platforms for dynamic multithreading do not provide repeatable parallel random-number generators. This paper proposes that a mechanism called pedigrees be built into the runtime system to enable efficient deterministic parallel randomnumber generation. Experiments with the open-source MIT Cilk runtime system show that the overhead for maintaining pedigrees is negligible. Specifically, on a suite of 10 benchmarks, the relative overhead of Cilk with pedigrees to the original Cilk has a geometric mean of less than 1%.We persuaded Intel to modify its commercial C/C++ compiler, which provides the Cilk Plus concurrency platform, to include pedigrees, and we built a library implementation of a deterministic parallel random-number generator called DOTMIX that compresses the pedigree and then "RC6-mixes" the result. The statistical quality of DOTMIX is comparable to that of the popular Mersenne twister, but somewhat slower than a nondeterministic parallel version of this efficient and high-quality serial random-number generator. The cost of calling DOTMIX depends on the "spawn depth" of the invocation. For a naive Fibonacci calculation with n = 40 that calls DOTMIX in every node of the computation, this "price of determinism" is about a factor of 2.3 in running time over the nondeterministic Mersenne twister, but for more realistic applications with less intense use of random numbers -such as a maximalindependent-set algorithm, a practical samplesort program, and a Monte Carlo discrete-hedging application from QuantLib -the observed "price" was at most 21%, and sometimes much less. Moreover, even if overheads were several times greater, applications using DOTMIX should be amply fast for debugging purposes, which is a major reason for desiring repeatability.

Random numbers for parallel computers: Requirements and methods, with emphasis on GPUs

“…A fast counter-based RNG for GPUs is proposed for that using MD5 in [78] and faster ones using TEA in [80]. Other ones that can fit in one thread in a traditional (discrete) GPU can be found in [8,49,59,69].…”

“…But this often does not match the need of users, e.g., if one wishes to run one simulation thread per PE. RNG frameworks and implementations specifically adapted to GPUs, and other parallel processing environments such as programmable arrays, have also been proposed in recent years; see for example [6,19,49,58,65,64,73,74,75,80]. Parallel processing accelerators can be exploited in different ways in a simulation.…”

Splittable pseudorandom number generators using cryptographic hashing

“…Leiserson et al [26] propose a PRNG for deterministic parallel random number generation. Their generator is integrated with the MIT Cilk platform, which tracks the call path of the current program location, which is then used to generate random numbers in a way that is independent of thread scheduling.…”

“…Moreover, splittable PRNGs are essential when generating random infinite values, such as random infinite lists in a lazy language, or random functions. In addition, deterministic distribution of parallel random number streams, which is of interest to the High-Performance Computing community [22,26], can be realised using splitting.…”

“…An efficient PRNG that supports splitting under the hood has been proposed by Leiserson et al [26]. To generate random numbers, the generator hashes a path, which identifies the current program location, using a hash function.…”