Using memory mapping to support cactus stacks in work-stealing runtime systems

Abstract

Many multithreaded concurrency platforms that use a workstealing runtime system incorporate a "cactus stack," wherein a function's accesses to stack variables properly respect the function's calling ancestry, even when many of the functions operate in parallel. Unfortunately, such existing concurrency platforms fail to satisfy at least one of the following three desirable criteria:• full interoperability with legacy or third-party serial binaries that have been compiled to use an ordinary linear stack, • a scheduler that provides near-perfect linear speedup on applications with sufficient parallelism, and • bounded and efficient use of memory for the cactus stack. We have addressed this cactus-stack problem by modifying the Linux operating system kernel to provide support for thread-local memory mapping (TLMM). We have used TLMM to reimplement the cactus stack in the open-source Cilk-5 runtime system. The Cilk-M runtime system removes the linguistic distinction imposed by Cilk-5 between serial code and parallel code, erases Cilk-5's limitation that serial code cannot call parallel code, and provides full compatibility with existing serial calling conventions. The Cilk-M runtime system provides strong guarantees on scheduler performance and stack space. Benchmark results indicate that the performance of the prototype Cilk-M 1.0 is comparable to the Cilk 5.4.6 system, and the consumption of stack space is modest.

A scalable and composable map-reduce system

“…Nonetheless, the internal data structure size grows moderately as the thread count increases by about 18 KB per thread for CilkMR and about 1 KB per thread for Phoenix++. This difference arises because CilkMR requires a varying number of stacks depending on how work stealing progresses [21].…”

Memory-mapping support for reducer hyperobjects

“…This section describes thread-local memory mapping (TLMM) [22,23], which Cilk-M uses to cause the virtual-memory hardware to map reducers to local views. TLMM provides an efficient and flexible way for a thread to map certain regions of virtual memory independently from other threads while still sharing most of its virtual-memory address space.…”

“…Cilk-M's TLMM mechanism [22,23] was originally developed to enable a work-stealing runtime system to maintain a "cactusstack" abstraction, thereby allowing arbitrary calling between parallel and serial code. To implement TLMM, we modified the Linux kernel, producing a kernel version we call TLMM-Linux.…”

“…The operating-system support employs thread-local memory mapping (TLMM) [23]. TLMM enables the virtual-memory The relative overhead for ordinary L1-cache memory accesses, memory-mapped reducers, hypermap reducers, and locking.…”

“…This support for efficient view transferal allows workers to perform reductions without extra memory mapping. We implemented our memory-mapping strategy by modifying Cilk-M [23], a Cilk runtime system that employs TLMM to manage the "cactus stack," to make it a plug-in replacement for Intel's Cilk Plus runtime system. That is, we modified the Cilk-M runtime system to replace the native Cilk runtime system shipped with Intel's C++ compiler by making Cilk-M conform to the Intel Cilk Plus Application Binary Interface (ABI) [17].…”