A statistical analysis of lossily compressed climate model data

Poppick, Andrew; Nardi, Joseph; Feldman, Noah; Baker, Allison H.; Pinard, Alexander; Hammerling, Dorit

doi:10.1016/j.cageo.2020.104599

Cited by 27 publications

(15 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Today's lossy compression techniques have been used in many HPC scientific applications for saving storage space and reducing the I/O cost of saving data [14,30,40]. In this paper we focus on utilizing SZ lossy compression for cosmological simulation with consideration of specified analysis error control.…”

Section: Lossy Compression For Scientific Datamentioning

confidence: 99%

Adaptive Configuration of In Situ Lossy Compression for Cosmology Simulations via Fine-Grained Rate-Quality Modeling

Jin

Pulido

Grosset

et al. 2021

Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing

View full text Add to dashboard Cite

Extreme-scale cosmological simulations have been widely used by today's researchers and scientists on leadership supercomputers. A new generation of error-bounded lossy compressors has been used in workflows to reduce storage requirements and minimize the impact of throughput limitations while saving large snapshots of high-fidelity data for post-hoc analysis. In this paper, we propose to adaptively provide compression configurations to compute partitions of cosmological simulations with newly designed postanalysis aware rate-quality modeling. The contribution is fourfold:(1) We propose a novel adaptive approach to select feasible error bounds for different partitions, showing the possibility and efficiency of adaptively configuring lossy compression for each partition individually. (2) We build models to estimate the overall loss of post-analysis result due to lossy compression and to estimate compression ratio, based on the property of each partition.(3) We develop an efficient optimization guideline to determine the best-fit configuration of error bounds combination in order to maximize the compression ratio under acceptable post-analysis quality loss. (4) Our approach introduces negligible overheads for feature extraction and error-bound optimization for each partition, enabling post-analysis-aware in situ lossy compression for cosmological simulations. Experiments show that our proposed models are highly accurate and reliable. Our fine-grained adaptive configuration approach improves the compression ratio of up to 73% on the tested datasets with the same post-analysis distortion with only 1% performance overhead.

show abstract

Section: Lossy Compression For Scientific Datamentioning

confidence: 99%

Adaptive Configuration of In Situ Lossy Compression for Cosmology Simulations via Fine-Grained Rate-Quality Modeling

Jin

Pulido

Grosset

et al. 2021

Proceedings of the 30th International Symposium on High-Performance Parallel and Distributed Computing

View full text Add to dashboard Cite

show abstract

“…On the other hand, round-off, truncation, iteration and model errors that appear in numerical simulations also have limited precision/accuracy. Thus, using lossy compression techniques to approximate/compress floating-point data is acceptable and one of the best solutions for solving the big scientific data issue [19], [28], [29].…”

Section: Lossy Compression Fault Modelmentioning

confidence: 99%

LCFI: A Fault Injection Tool for Studying Lossy Compression Error Propagation in HPC Programs

Shan

Shamji

Tian

et al. 2020

Preprint

View full text Add to dashboard Cite

Error-bounded lossy compression is becoming more and more important to today's extreme-scale HPC applications because of the ever-increasing volume of data generated. Errorbounded lossy compressors have been widely used in in situ visualization, data stream intensity reduction, storage reduction, I/O performance improvement, checkpoint/restart acceleration, memory footprint reduction, etc.Although many works have optimized ratio, quality, and performance for different error-bounded lossy compressors, there is none of the existing works attempting to systematically understand the impact of lossy compression errors on HPC application due to error propagation.In this paper, we propose and develop a lossy compression fault injection tool, called LCFI. To the best of our knowledge, this is the first tool that help both lossy compressor developers and users to systematically and comprehensively understand the impact of lossy compression errors on any given HPC programs.The contributions of this work are threefold.(1) We propose an efficient approach to inject lossy compression faults according to a statistical analysis of compression errors for different stateof-the-art error-bounded lossy compressors. (2) We build a fault injector which is highly applicable, customizable, ease-to-use in generating top-down comprehensive results. We use a simple sample program to demonstrate the use of LCFI. (3) We evaluate LCFI on four representative HPC benchmark programs with different lossy compression errors and derive several important insights based on our observations.

show abstract

“…The notion of "compression error" is not well-defined, but needs to be specified in view of a particular application. The impact of compression on data analysis has been studied empirically [30,41] and statistically [42,43]. Ideally, the compression scheme is tailored towards the desired error metric.…”

Section: Error Metrics and Error Propagation Affect Compression Accur...mentioning

confidence: 99%

Compression Challenges in Large Scale Partial Differential Equation Solvers

Götschel

Weiser

2019

Algorithms

View full text Add to dashboard Cite

Solvers for partial differential equations (PDE) are one of the cornerstones of computational science. For large problems, they involve huge amounts of data that needs to be stored and transmitted on all levels of the memory hierarchy. Often, bandwidth is the limiting factor due to relatively small arithmetic intensity, and increasingly so due to the growing disparity between computing power and bandwidth. Consequently, data compression techniques have been investigated and tailored towards the specific requirements of PDE solvers during the last decades. This paper surveys data compression challenges and discusses examples of corresponding solution approaches for PDE problems, covering all levels of the memory hierarchy from mass storage up to main memory. Exemplarily, we illustrate concepts at particular methods, and give references to alternatives.

show abstract

A statistical analysis of lossily compressed climate model data

Cited by 27 publications

References 16 publications

Adaptive Configuration of In Situ Lossy Compression for Cosmology Simulations via Fine-Grained Rate-Quality Modeling

Adaptive Configuration of In Situ Lossy Compression for Cosmology Simulations via Fine-Grained Rate-Quality Modeling

LCFI: A Fault Injection Tool for Studying Lossy Compression Error Propagation in HPC Programs

Compression Challenges in Large Scale Partial Differential Equation Solvers

Contact Info

Product

Resources

About