A smart hybrid content-defined chunking algorithm for data deduplication in cloud storage

Abstract

The enormous growth of data requires a large amount of storage space in the cloud server, which occupies mostly by the redundant data. The deduplication technique avoids redundancy to utilize cloud storage effectively. Chunking is the process to break the data into chunks, determine duplicates. Many algorithms exist, to handle the deduplication efficiency, reducing the chunk variance, and improving the computational overhead continues to be a challenging task. To solve the above challenge, we propose a Smart Chunker (SC) algorithm, which operates with hybrid chunking based on file size as File Level and Content Defined Chunking (CDC). File-level Chunking is assigned only for less than 2KB file size, and the exceeding file size falls with CDC. We aim Optimus Prime Chunking (OPC) in CDC to break the chunks with prime numbers and involves a minimum number of comparisons with fewer conditions and low computation overhead. This work reduces the processing time without hash and window for computation. Thus, it provides a constant average chunk size to distribute the equal chunk variance in the cloud storage with a 17% reduction in chunk time. Our OPC result attains high throughput of more than 3.3x, compared to other CDC algorithms.