A study on data deduplication in HPC storage systems

Abstract

Deduplication is a storage saving technique that is highly successful in enterprise backup environments. On a file system, a single data block might be stored multiple times across different files, for example, multiple versions of a file might exist that are mostly identical. With deduplication, this data replication is localized and redundancy is removed – by storing data just\ud once, all files that use identical regions refer to the same unique data. The most common approach splits file data into chunks\ud and calculates a cryptographic fingerprint for each chunk. By checking if the fingerprint has already been stored, a chunk is classified as redundant or unique. Only unique chunks are stored. This paper presents the first study on the potential of data deduplication in HPC centers, which belong to the most demanding storage producers. We have quantitatively assessed this potential for capacity reduction for 4 data centers (BSC, DKRZ,\ud RENCI, RWTH). In contrast to previous deduplication studies focusing mostly on backup data, we have analyzed over one PB\ud (1212 TB) of online file system data. The evaluation shows that typically 20% to 30% of this online data can be removed by applying data deduplication techniques, peaking up to 70% for some data sets. This reduction can only be achieved by a subfile deduplication approach, while approaches based on whole-file\ud comparisons only lead to small capacity savings.Peer ReviewedPostprint (published version

Deduplication Potential of HPC Applications’ Checkpoints

“…The zero chunk contributes significantly to the deduplication potential in enterprise backups and virtual machine images [55], [56]. In their HPC study, Meister et al found that between 3.1% and 24.3% of their HPC data consist of zero chunks [12]. In our case, the zero chunk is the most used chunk and is the main source of redundant data for every application and chunk size, except CDC with an average chunk size of 32 KB.…”

“…However, we vary the number of used processes in Section V-C. Table I shows the different sizes of the checkpoints. c) Deduplication: We analyzed each checkpoint with the FS-C deduplication tool suite [49], which has already been applied in several deduplication studies [50], [51]. We chose fixed-sized chunking and content-defined chunking (CDC) as chunking methods.…”

“…Besides backup, primary storage also provides deduplication potential. Meister et al showed a huge potential for data deduplication in HPC storage systems that is not facilitated by today's HPC file systems [12]. Nicolae and Kulkarni et al applied the deduplication approach to the checkpointing use case and discussed systems to reduce the I/O load during checkpointing [13], [14].…”

Exascale Storage Systems - An Analytical Study of Expenses

“…Instead of storing duplicate data, a reference to the original block is created for each repeated occurrence. Our previously conducted study for HPC data already showed great potential for data savings, allowing 20-30 % of redundant data to be eliminated on average [21]. To determine the potential savings, we independently scanned 12 sets of directories with a total amount of data of more than 1 PB.…”

Towards Scalable Checkpoint Restart: A Collective Inline Memory Contents Deduplication Proposal

“…attempt reduce the checkpoint sizes. While there are several techniques proposed in this direction, recent studies [6] point out that deduplication (i.e. identifying identical copies of data and storing only one copy) shows promising potential, with reported reductions of up to 70%.…”