Improving performance of iterative methods by lossy checkponting

Abstract

Iterative methods are commonly used approaches to solve large, sparse linear systems, which are fundamental operations for many modern scientific simulations. When the large-scale iterative methods are running with a large number of ranks in parallel, they have to checkpoint the dynamic variables periodically in case of unavoidable fail-stop errors, requiring fast I/O systems and large storage space. To this end, significantly reducing the checkpointing overhead is critical to improving the overall performance of iterative methods. Our contribution is fourfold. (1) We propose a novel lossy checkpointing scheme that can significantly improve the checkpointing performance of iterative methods by leveraging lossy compressors. (2) We formulate a lossy checkpointing performance model and derive theoretically an upper bound for the extra number of iterations caused by the distortion of data in lossy checkpoints, in order to guarantee the performance improvement under the lossy checkpointing scheme. (3) We analyze the impact of lossy checkpointing (i.e., extra number of iterations caused by lossy checkpointing files) for multiple types of iterative methods. (4) We evaluate the lossy checkpointing scheme with optimal checkpointing intervals on a high-performance computing environment with 2,048 cores, using a well-known scientific computation package PETSc and a state-of-the-art checkpoint/restart toolkit. Experiments show that our optimized lossy checkpointing scheme can significantly reduce the fault tolerance overhead for iterative methods by 23%∼70% compared with traditional checkpointing and 20%∼58% compared with lossless-compressed checkpointing, in the presence of system failures.

H-RADIC: A Fault Tolerance Framework for Virtual Clusters on Multi-Cloud Environments

“…Implement a mechanism that accelerates the transfer rate, by improving I/O, use incremental checkpoints and/or compress [16] the checkpoints even more.…”

Use cases of lossy compression for floating-point data in scientific data sets

“…Second, the performance degradation due to the extra application iterations must be mitigated by the time saved on I/O and storage operations. Third, in order to optimize the overall execution performance in the presence of failures, one needs to calculate the checkpoint intervals based on the revised performance model proposed in (Tao et al, 2018). According to the model, the compression and decompression speeds need to be able to be estimated based on the (compressed) checkpoint size and user-set compression error bound.…”

“…For demonstration purposes, we use the sparse matrix arising from discretizing a 3-D Poisson’s equation. We refer readers to (Tao et al, 2018) for the matrix details. We use the PETSc (v3.8) (Balay et al, 2018) library for GMRES and its default preconditioner (block Jacobi with ILU/IC).…”

“…GMRES: generalized minimal residual; CG: conjugate gradient. Extracted from our prior conference paper (Tao et al, 2018).…”

“…Average time of one checkpoint and recovery for GMRES with different checkpointing techniques (extracted from our prior conference paper(Tao et al, 2018)). (a) Checkpoint and (b) recovery.…”