Virtual chunks: On supporting random accesses to scientific data in compressible storage systems

Abstract

Abstract-Data compression could ameliorate the I/O pressure of scientific applications on high-performance computing systems. Unfortunately, the conventional wisdom of naively applying data compression to the file or block brings the dilemma between efficient random accesses and high compression ratios. Filelevel compression can barely support efficient random accesses to the compressed data: any retrieval request need trigger the decompression from the beginning of the compressed file. Block-level compression provides flexible random accesses to the compressed data, but introduces extra overhead when applying the compressor to each every block that results in a degraded overall compression ratio. This paper introduces a concept called virtual chunks aiming to support efficient random accesses to the compressed scientific data without sacrificing its compression ratio. In essence, virtual chunks are logical blocks identified by appended references without breaking the physical continuity of the file content. These additional references allow the decompression to start from an arbitrary position (efficient random access), and retain the file's physical entirety to achieve high compression ratio on par with file-level compression. One potential concern of virtual chunks lies on its space overhead (from the additional references) that degrades the compression ratio, but our analytic study and experimental results demonstrate that such overhead is negligible. We have implemented virtual chunks in two forms: a middleware to the GPFS parallel file system, and a module in the FusionFS distributed file system. Large-scale evaluations on up to 1,024 cores showed that virtual chunks could help improve the I/O throughput by 2X speedup.

Towards cost-effective and high-performance caching middleware for distributed systems

“…In future, we will integrate the proposed caching mechanism to other systems such as file compression [62,63], data provenance [64,65] and job scheduling [66]. We plan to further investigate the tradeoff between performance (for example, GPU acceleration [67]) and cost (for example, scientific applications on EC2 [68]) with the introduction of memoryclass cache, and explore the viability to extend the current approach into incremental mechanisms [69][70][71].…”

A convergence of key‐value storage systems from clouds to supercomputers

“…This section presents some real systems that have adopted ZHT as a building block. It also leads to additional publications .…”

Exploiting multi‐cores for efficient interchange of large messages in distributed systems

“…It is not uncommon to deal with large objects in a distributed system. In scientific computing, for example as our prior work showed, datasets were so large that they were usually compressed to be serialized onto the hard disk. As another example, in our previous work on implementing a distributed file system , we showed that a large directory composed of thousands of small files could result in a huge metadata blob stored in a distributed key‐value store .…”