Efficient Checkpointing of Loop-Based Codes for Non-volatile Main Memory

ACM Trans. Archit. Code Optim.

Elnawawy

Elkhouly

et al. 2019

Self Cite

Future main memory will likely include Non-Volatile Memory. Non-Volatile Main Memory (NVMM) provides an opportunity to rethink checkpointing strategies for providing failure safety to applications. While there are many checkpointing and logging schemes in the literature, their use must be revisited as they incur high execution time overheads as well as a large number of additional writes to NVMM, which may significantly impact write endurance.In this article, we propose a novel recompute-based failure safety approach and demonstrate its applicability to loop-based code. Rather than keeping a fully consistent logging state, we only log enough state to enable recomputation. Upon a failure, our approach recovers to a consistent state by determining which parts of the computation were not completed and recomputing them. Effectively, our approach removes the need to keep checkpoints or logs, thus reducing execution time overheads and improving NVMM write endurance at the expense of more complex recovery. We compare our new approach against logging and checkpointing on five scientific workloads, including tiled matrix multiplication, on a computer system model that was built on gem5 and supports Intel PMEM instruction extensions. For tiled matrix multiplication, our recompute approach incurs an execution time overhead of only 5%, in contrast to 8% overhead with logging and 207% overhead with checkpointing. Furthermore, recompute only adds 7% additional NVMM writes, compared to 111% with logging and 330% with checkpointing. We also conduct experiments on real hardware, allowing us to run our workloads to completion while varying the number of threads used for computation. These experiments substantiate our simulation-based observations and provide a sensitivity study and performance comparison between the Recompute Scheme and Naive Checkpointing. INTRODUCTIONNon-volatile memory (NVM) technologies have been advancing rapidly, and some of them are a strong contender for use as a future main memory, either for augmenting or replacing DRAM. One such an example is 3D Xpoint memory, which will be brought to market in 2017 by Intel and Micron [28]. These new non-volatile main memory (NVMM) technologies are byte-addressable and have access latencies that are not much slower than DRAM [3,6,8,28,34,37,38,50]. NVMMs are expected to have a limited write endurance, making it imperative to keep the number of writes low [7]. Despite the limited write endurance and relatively high write latency compared to DRAM, NVMM's density and cost advantage over DRAM and near-zero idle power consumption make them a compelling candidate to replace or augment DRAM in high-performance computers [5].At the same time, as high-performance computing (HPC) relies on an increasing number of nodes and components, it becomes increasingly likely that long-running computation will be interrupted by failures before completing. Frequent checkpointing has become essential because it allows applications to resume from a recent snapshot rather than re-execute from ...

Section: Durable Transactions With Write Ahead Loggingmentioning

confidence: 99%

Section: Our Recompute Approachmentioning

confidence: 99%

Efficient Checkpointing with Recompute Scheme for Non-volatile Main Memory

ACM Trans. Archit. Code Optim.

Elnawawy

Elkhouly

et al. 2019

Self Cite

“…Through either redo or undo logging, they enable data to be recovered if a failure occurs during a transaction. The overhead of logging can be significant and there are some recent works to either reduce the size of the logs using recomputation [3] or to improve the performance using hardware logging [17,18].…”

Section: Memory Persistency Models On Cpusmentioning

confidence: 99%

“…Persistent data storage in main memory provides an opportunity to achieve recoverable data structures (RDS), which allows programs to recover from crashes just by using data in main memory instead of a checkpoint. Recent research [3,4] showed that by relying on RDS instead of checkpoints, highly significant performance and write endurance improvements can be obtained. Achieving RDS requires persistency models along with instruction support, and a crash recovery technique such as logging.…”

Section: Introductionmentioning

confidence: 99%

Exploring Memory Persistency Models for GPUs

Lin

2019 28th International Conference on Parallel Architectures and Compilation Techniques (PACT)

Solihin

et al. 2019

Self Cite

Given its high integration density, high speed, byte addressability, and low standby power, nonvolatile or persistent memory is expected to supplement/replace DRAM as main memory. Through persistency programming models (which define durability ordering of stores) and durable transaction constructs, the programmer can provide recoverable data structure (RDS) which allows programs to recover to a consistent state after a failure. While persistency models have been well studied for CPUs, they have been neglected for graphics processing units (GPUs). Considering the importance of GPUs as a dominant accelerator for high performance computing, we investigate persistency models for GPUs. GPU applications exhibit substantial differences with CPUs applications, hence in this paper we adapt, re-architect, and optimize CPU persistency models for GPUs. We design a pragma-based compiler scheme to express persistency models for GPUs. We identify that the thread hierarchy in GPUs offers intuitive scopes to form epochs and durable transactions. We find that undo logging produces significant performance overheads. We propose to use idempotency analysis to reduce both logging frequency and the size of logs. Through both real-system and simulation evaluations, we show low overheads of our proposed architecture support.

“…Meanwhile, a checksum is calculated to discover persistence errors in NVM and recompute inconsistent results. In References [2,17], recomputing is used for a chunk of data whose consistency is questionable. Unlike lazy persistency, NVM instructions are inserted to force data flushing from cache to persistent memory periodically.…”

Section: Introductionmentioning

confidence: 99%

Compiler-support for Critical Data Persistence in NVM

Elkhouly

ACM Trans. Archit. Code Optim.

Hayashi

et al. 2019

Self Cite

Non-volatile Main Memories (NVMs) offer a promising way to preserve data persistence and enable computation recovery in case of failure. While the use of NVMs can significantly reduce the overhead of failure recovery, which is the case with High-Performance Computing (HPC) kernels, rewriting existing programs or writing new applications for NVMs is non-trivial. In this article, we present a compiler-support that automatically inserts complex instructions into kernels to achieve NVM data-persistence based on a simple programmer directive. Unlike checkpointing techniques that store the whole system state, our technique only persists user-designated objects as well as some parameters required for safe recovery such as loop induction variables. Also, our technique can reduce the number of data transfer operations, because our compiler coalesces consecutive memory-persisting operations into a single memory transaction per cache line when possible. Our compiler-support is implemented in the LLVM tool-chain and introduces the necessary modifications to loop-intensive computational kernels (e.g., TMM, LU, Gauss, and FFT) to force data persistence. The experiments show that our proposed compiler-support outperforms the most recent checkpointing techniques while its performance overheads are insignificant. CCS Concepts: • Software and its engineering → Source code generation; Runtime environments; Main memory; • Hardware → Emerging languages and compilers;