Scalable management of storage for massive quality-adjustable sensor data

Lee, Dong‐Eun; Ryu, Jee-Youl; Shin, Hyundong

doi:10.1007/s00607-015-0465-6

Cited by 3 publications

(1 citation statement)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It has been studied that full utilization of correlation yields better compression ratios than partial utilization does [14,13]. On 1D arrays, though ZFP performs significantly better than other methods, its compression ratios are lower than on 3D arrays, as shown in Fig.…”

Section: Zfp Floating-point Compressionmentioning

confidence: 99%

Novel Data Reduction Based on Statistical Similarity

Lee

Sim

Choi

et al. 2016

Proceedings of the 28th International Conference on Scientific and Statistical Database Management

Self Cite

View full text Add to dashboard Cite

Applications such as scientific simulations and power grid monitoring are generating so much data quickly that compression is essential to reduce storage requirement or transmission capacity. To achieve better compression, one is often willing to discard some repeated information. These lossy compression methods are primarily designed to minimize the Euclidean distance between the original data and the compressed data. But this measure of distance severely limits either reconstruction quality or compression performance. We propose a new class of compression method by redefining the distance measure with a statistical concept known as exchangeability. This approach reduces the storage requirement and captures essential features, while reducing the storage requirement. In this paper, we report our design and implementation of such a compression method named IDEALEM. To demonstrate its e↵ectiveness, we apply it on a set of power grid monitoring data, and show that it can reduce the volume of data much more than the best known compression method while maintaining the quality of the compressed data. In these tests, IDEALEM captures extraordinary events in the data, while its compression ratios can far exceed 100.

show abstract

Section: Zfp Floating-point Compressionmentioning

confidence: 99%