Exploring Lossy Compressibility through Statistical Correlations of Scientific Datasets

Krasowska, David; Bessac, Julie; Underwood, Robert; Calhoun, Jon C.; Di, Sheng; Cappello, Franck

doi:10.1109/drbsd754563.2021.00011

Cited by 8 publications

(10 citation statements)

References 21 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, a few works have investigated spatial and temporal statistical predictors of CR. In Krasowska et al (2021), global and local spatial correlation ranges that are estimated via a variogram are explored as candidate statistical predictors for CR. No CR-prediction setup is proposed, however; and, as shown in Figure 4, the spatial correlation on its own is insufficient to fully characterize 2D slices in terms of CR.…”

Section: Existing Methods For Cr Predictionmentioning

confidence: 99%

“…Gaussian samples of type 4 are the most challenging because correlation scales are picked randomly and aggregated with spatial weights creating samples with strong spatial heterogeneity that may not be encountered in most continuous scientific simulations. These samples challenge the chosen statistical predictors and highlight the need to account for heterogeneous multiscale information, as pointed out by Krasowska et al (2021). SZ shows higher errors for samples of Type 1 than for the other sample types, this is due to the fact that CRs observed in Type 1 samples have a much wider range than the other ones, hence larger prediction errors.…”

Section: Cr Predictions For Different Types Of Gaussian Samplesmentioning

confidence: 96%

“…In particular, the stronger the association with neighboring bits, the greater the BIR. In Krasowska et al (2021) The SVD of a m × n-matrix X provides the following decomposition: X = UΣV* with U an m × m complex unitary (UU* = I m ) matrix (left singular vectors), Σ an m × n rectangular diagonal matrix with non-negative real numbers on the diagonal (diagonal entries σ i = Σ ii are the singular values), and V an n × n complex unitary (VV* = I n ) matrix (right singular vectors). We consider the percentage of singular values needed to recover 99% of the total variance of X, which is proportional to P minðn, mÞ i¼1 σ 2 i , when X has Tucker (1963) for tensors and is seen as the extension of SVD to a higher-order tensor context Kolda and Bader (2009).…”

Section: Statistical Predictors Of Compression Ratiosmentioning

confidence: 99%

“…This is the first formulation of a generic (compressor-free) statistical prediction model that uses training only from observed compression ratios for its specialization to a compressor. The generic model can specialize to any lossy compressor before prediction, as opposed to other prediction models that depend on knowledge of a compressor’s design principles Lu et al (2018); Qin et al (2020); Krasowska et al (2021) or need several trial runs during prediction Underwood et al (2020). We note that all existing and currently proposed methods require the use of compressors to generate compression ratios as training data.…”

Section: Introductionmentioning

confidence: 99%

“…(ii) We study leading state-of-the-art lossy compressors covering different compression models with different principles. (iii) We carefully investigate various responses of our developed compression ratio predictions to the different data compressor-prediction schemes used in the state-of-the-art lossy compressor SZ, providing a more in-depth understanding of the compression quality of SZ2 compared with the prior works Qin et al (2020); Krasowska et al (2021) that studied SZ2. SZ is widely used in the community, but its compression ratio remains challenging to predict because of its multiprediction-scheme approach.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

Black-box statistical prediction of lossy compression ratios for scientific data

Underwood

Bessac

Krasowska

et al. 2023

The International Journal of High Performance Computing Applica

Self Cite

View full text Add to dashboard Cite

Lossy compressors are increasingly adopted in scientific research, tackling volumes of data from experiments or parallel numerical simulations and facilitating data storage and movement. In contrast with the notion of entropy in lossless compression, no theoretical or data-based quantification of lossy compressibility exists for scientific data. Users rely on trial and error to assess lossy compression performance. As a strong data-driven effort toward quantifying lossy compressibility of scientific datasets, we provide a statistical framework to predict compression ratios of lossy compressors. Our method is a two-step framework where (i) compressor-agnostic predictors are computed and (ii) statistical prediction models relying on these predictors are trained on observed compression ratios. Proposed predictors exploit spatial correlations and notions of entropy and lossyness via the quantized entropy. We study 8+ compressors on 6 scientific datasets and achieve a median percentage prediction error less than 12%, which is substantially smaller than that of other methods while achieving at least a 8.8× speedup for searching for a specific compression ratio and 7.8× speedup for determining the best compressor out of a collection.

show abstract

Section: Existing Methods For Cr Predictionmentioning

confidence: 99%

Section: Cr Predictions For Different Types Of Gaussian Samplesmentioning

confidence: 96%

Section: Statistical Predictors Of Compression Ratiosmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Black-box statistical prediction of lossy compression ratios for scientific data

Underwood

Bessac

Krasowska

et al. 2023

The International Journal of High Performance Computing Applica

Self Cite

View full text Add to dashboard Cite

show abstract

Discussion on “Saving Storage in Climate Ensembles: A Model-Based Stochastic Approach”

Bessac

Underwood

2023

JABES

View full text Add to dashboard Cite

We thank the authors for this interesting paper that highlights important ideas and concepts for the future of climate model ensembles and their storage, as well as future uses of stochastic emulators. Stochastic emulators are particularly relevant because of the statistical nature of climate model ensembles, as discussed in previous work of the authors (Castruccio et al. in J Clim 32:8511–8522, 2019; Hu and Castruccio in J Clim 34:8409–8418, 2021). We thank the authors for sharing of some of their data with us in order to illustrate this discussion. In the following, in Sect. 1 we discuss alternative techniques currently used and studied, namely lossy compression and ideas emerging from the climate modeling community, that could feed the discussion on ensemble and storage. In that section, we also present numerical results of compression performed on the data shared by the authors. In Sect. 2, we discuss the current statistical model proposed by the authors and its context. We discuss other potential uses of stochastic emulators in climate and Earth modeling.

show abstract

Runway: In-transit Data Compression on Heterogeneous HPC Systems

Ravi

Byna

Becchi

2023

2023 IEEE/ACM 23rd International Symposium on Cluster, Cloud and Internet Computing (CCGrid)

View full text Add to dashboard Cite

To alleviate bottlenecks in storing and accessing data on high-performance computing (HPC) systems, I/O libraries are enabling computation while data is in-transit, such as HDF5 filters. For scientific applications that commonly use floatingpoint data, error-bounded lossy compression methods are a critical technique to significantly reduce the storage and bandwidth requirements. Thus far, deciding when and where to schedule in-transit data transformations, such as compression, has been outside the scope of I/O libraries.In this paper, we introduce Runway, a runtime framework that enables computation on in-transit data with an object storage abstraction. Runway is designed to be extensible to execute userdefined functions at runtime. In this effort, we focus on studying methods to offload data compression operations to available processing units based on latency and throughput. We compare the performance of running compression on multi-core CPUs, as well as offloading it to a GPU and a Data Processing Unit (DPU). We implement a state-of-the-art error-bounded lossy compression algorithm, SZ3, as a Runway function with a variant optimized for DPUs. We propose dynamic modeling to guide scheduling decisions for in-transit data compression. We evaluate Runway using four scientific datasets from the SDRBench benchmark suite on a the Perlmutter supercomputer at NERSC.

show abstract

Exploring Lossy Compressibility through Statistical Correlations of Scientific Datasets

Cited by 8 publications

References 21 publications

Black-box statistical prediction of lossy compression ratios for scientific data

Black-box statistical prediction of lossy compression ratios for scientific data

Discussion on “Saving Storage in Climate Ensembles: A Model-Based Stochastic Approach”

Runway: In-transit Data Compression on Heterogeneous HPC Systems

Contact Info

Product

Resources

About