Noise-Tolerant Explicit Stencil Computations for Nonuniform Process Execution Rates

Abstract

Next-generation HPC computing platforms are likely to be characterized by significant, unpredictable nonuniformities in execution time among compute nodes and cores. The resulting load imbalances from this nonuniformity are expected to arise from a variety of sources-manufacturing discrepancies, dynamic power management, runtime component failure, OS jitter, software-mediated resiliency, and TLB/-cache performance variations, for example. It is well understood that existing algorithms with frequent points of bulk synchronization will perform relatively poorly in the presence of these sources of process nonuniformity. Thus, recasting classic bulk synchronous algorithms into more asynchronous, coarse-grained parallelism is a critical area of research for next-generation computing. We propose a class of parallel algorithms for explicit stencil computations that can tolerate these nonuniformities by decoupling per process communication and computation in order for each process to progress asynchronously while maintaining solution correctness. These algorithms are benchmarked with a 1D domain decomposed ("slabbed") implementation of the 2D heat equation as a model problem, and are tested in the presence of simulated nonuniform process execution rates. The resulting performance is compared to a classic bulk synchronous implementation of the model problem. Results show that the runtime of this article's algorithm on a machine with simulated process nonuniformities is 5-99% slower than the runtime of its classic counterpart on a machine free of nonuniformities. However, when both algorithms are run on a machine with comparable synthetic process nonuniformities, this article's algorithm is 1-37 times faster than its classic counterpart.

Taming Energy Consumption Variations In Systems Benchmarking

“…In [14], the authors introduced a variation-aware algorithm that improves application performance under a power constraint by determining module-level (individual processor and associated DRAM) power allocation, with up to 5.4× speedup. The authors of [12] proposed parallel algorithms that tolerate the variability and the non-uniformity by decoupling per process communication over the available CPU. Acun et al [2] found out a way to reduce the energy variation on Ivy Bridge and Sandy Bridge processors, by disabling the Turbo Boost feature to stabilize the execution time over a set of processors.…”

MOANA: Modeling and Analyzing I/O Variability in Parallel System Experimental Design

“…Jitter in HPC is typically described as performance loss caused by the competition for resources between background processes and applications, or application interference [37,17]. Some OS jitter researchers have simulated these effects at scale [9], while others have proposed applications [12] or systems [4] that can account for these types of variability. Additionally, schedulers are often identified as causes of significant variability in HPC systems [32] and might be classified as a form of jitter.…”

“…Such variability is cited as a significant barrier to exascale computing [29,18]. Unfortunately, variability (which includes OS jitter [21]) is both ubiquitous and elusive as its causes pervade and obscure performance across the systems stack from hardware [14,23] to middleware [20,1] to applications [12] to extreme-scale systems [29,35]. Additionally, as a number of recent reports attest [29,18], performance variability at scale can significantly reduce performance and energy efficiency [9,4].…”

An evaluation of the state of time synchronization on leadership class supercomputers

“…They demonstrated that by refactoring explicit stencil calculations, they were able to achieve speedups of 1 to 37 times faster than a traditional noise sensitive algorithm in an uncoordinated timing environment. 20 Beyond HPC, the impact of timing to more general scenarios in concurrency is also interesting. Google's Spanner uses coordinated time to achieve scalability and guarantee correctness of its distributed concurrency features such as externally consistent transactions, lock-free read-only transactions, and non-blocking reads.…”

“…In perhaps the largest impact reported, Hammouda et al studied a classic bulk synchronous implementation of an explicit stencil calculation in a range of environments spanning the absence of random detours to increasingly larger presence of random detours. They demonstrated that by refactoring explicit stencil calculations, they were able to achieve speedups of 1 to 37 times faster than a traditional noise sensitive algorithm in an uncoordinated timing environment …”