Concept and Analysis of Information Spaces to improve Prediction-Based Compression

Abstract

One of the scientific communities that generate the largest amounts of data today are the climate sciences. New climate models enable model integration at unprecedented resolution, simulating decades and centuries of climate change, including many complex interactions in the Earth system, under different scenarios. Previously, the CPU intensive numerical integration's used to be the bottleneck. Nowadays, limited storage space and ever increasing model output is the bigger challenge. The number of variables stored for post-processing analysis has to be limited to keep the data amounts small. For this reason, we look at lossless compression of climate data to make better use of available storage space. More specifically, we investigate prediction-based data compression. In prediction-based compression, data is processed in a predefined sequence. A prediction is provided for each data point based on prior data in the sequence. We show that there is a significant dependence of the compression ratio on the chosen traversal method and the underlying spatiotemporal data model. We examine the influence of this structural dependency on compression algorithms and explore possibilities to retrieve this information to improve compression ratios. To do this, we introduce the concept of Information Spaces (IS), which helps improve the predictions made by individual predictors by nearly 10% on average. More importantly, the standard deviation of the compression results is decreased by over 20% on average. The use of IS provides better predictions and more consistent compression ratios. Furthermore, it allows options for consolidation and fine-granular tuning of predictions, which are not possible with many common approaches used today.

Data Encoding in Lossless Prediction-Based Compression Algorithms

“…The shift can also help to increase throughput by using a simpler prediction model for compression and compensating for the weaknesses of the simpler prediction model with the shift. Table I shows that using a simple prediction model (Last Value) we can achieve a 50% higher throughput while still coming close to the LZC of the more sophisticated prediction method without shifted residual calculation 2 .…”

“…Since the traversal path and calculation of p is predefined by the compression algorithm, the value t can be reproduced by saving r. Although the calculation of a good predictive value p can be decisive for a good compression algorithm, this assessment is beyond the scope of this paper. Please refer to [2] for a comparison of prediction models. In this paper we study the influence of different encoding steps and assume that the same prediction model has been used by all algorithms.…”

“…In the last couple of years the development of compression algorithms for floating-point data experienced a renaissance. Several new methods were introduced [2]- [14] focusing on structural dependencies in gridded floating-point data. Most of these compression algorithms use a prediction-based compression.…”