Impact on Chunk Size on Deduplication and Disk Prefetch

Suzaki, Kuniyasu; Yagi, Toshiki; Kengo, Iijima; Artho, Cyrille; Watanabe, Yoshihito

doi:10.1007/978-3-642-25789-6_55

Cited by 1 publication

(1 citation statement)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, these expected chunk sizes lack either theoretical proof or experimental evaluation. Suzaki [21] studied the effect of chunk size from 4 KB to 512 KB on Input/Output (I/O) optimization by experiments, and showed large chunks (256 KB) were effective. However, the main concern of Suzaki's work is to optimize I/O rather than DER by setting chunk size, whereas, we focus on the impact of chunk size on DER in this paper.…”

Section: Related Workmentioning

confidence: 99%

A Logistic Based Mathematical Model to Optimize Duplicate Elimination Ratio in Content Defined Chunking Based Big Data Storage System

et al. 2016

View full text Add to dashboard Cite

Abstract:Deduplication is an efficient data reduction technique, and it is used to mitigate the problem of huge data volume in big data storage systems. Content defined chunking (CDC) is the most widely used algorithm in deduplication systems. The expected chunk size is an important parameter of CDC, and it influences the duplicate elimination ratio (DER) significantly. We collected two realistic datasets to perform an experiment. The experimental results showed that the current approach of setting the expected chunk size to 4 KB or 8 KB empirically cannot optimize DER. Therefore, we present a logistic based mathematical model to reveal the hidden relationship between the expected chunk size and the DER. This model provides a theoretical basis for optimizing DER by setting the expected chunk size reasonably. We used the collected datasets to verify this model. The experimental results showed that the R 2 values, which describe the goodness of fit, are above 0.9, validating the correctness of this mathematic model. Based on the DER model, we discussed how to make DER close to the optimum by setting the expected chunk size reasonably.

show abstract

Section: Related Workmentioning

confidence: 99%