The Design of Fast Content-Defined Chunking for Data Deduplication Based Storage Systems

Xia, Wen; Zou, Xiangyu; Jiang, Hong; Zhou, Yukun; Liu, Chuanyi; Feng, Dan; Hua, Yu; Hu, Yuchong; Zhang, Yucheng

doi:10.1109/tpds.2020.2984632

Cited by 53 publications

(20 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…FastCDC [11] uses four optimize techniques, resulting in a huge performance improvement in chunking for the deduplication system. As Fig.…”

Section: A Backgroundmentioning

confidence: 99%

“…To facilitate future study of CDC algorithms, we list essential properties that are indicators of a good or bad CDC algorithm, which were used by AE and FastCDC [11], [12]:…”

Section: Introductionmentioning

confidence: 99%

“…Compared to AE, MAXP algorithm needs to backtrack for comparison. FastCDC [11] uses several techniques to optimize the Rabin fingerprintbased CDC. As a result, FastCDC can outperform state-of-theart algorithm by 3-12× in chunking speed.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

UltraCDC:A Fast and Stable Content-Defined Chunking Algorithm for Deduplication-based Backup Storage Systems

Zhou

Wang

Xia

et al. 2022

2022 IEEE International Performance, Computing, and Communications Conference (IPCCC)

Self Cite

View full text Add to dashboard Cite

Content-Defined Chunking(CDC) is the key stage of data deduplication since it has a significant impact on deduplication system's throughput and deduplication efficiency. However, existing CDC algorithms suffer from high computation overhead, weak stability, and poor ability to handle low-entropy strings. In this paper, we propose UltraCDC, a fast and stable, high-efficiency deal with low-entropy strings, CDC algorithm for deduplication-based storage systems. There are four key techniques behind UltraCDC, namely, rolling compute boundary conditions, skipping sub-minimum chunk size, normalized chunking, and jumping to detect low-entropy strings. Using a sliding window to rolling compute boundary conditions not only accelerates the chunking stage but also makes it more resistant to boundary shift, the two techniques of skipping sub-minimum chunk size and normalized chunking can complement each other to speed up chunking without sacrificing deduplication ratio too much, and the jumping detection can detect more low-entropy strings than AE-opt2 without affecting chunking speed. We implemented UltraCDC in Destor, and the experimental results show that using the above four techniques, chunking speed is 1.5-10× faster than the state-of-the-art CDC approaches, while deduplication ratio is comparable or even higher than the classic Rabin-base CDC. In terms of the capability to detect low-entropy strings, UltraCDC is a CDC approach with the highest ability to detect low-entropy strings, 10 2 × and 2× higher than Rabinbased CDC and AE-opt2, respectively.

show abstract

“…FastCDC [11] uses four optimize techniques, resulting in a huge performance improvement in chunking for the deduplication system. As Fig.…”

Section: A Backgroundmentioning

confidence: 99%

“…To facilitate future study of CDC algorithms, we list essential properties that are indicators of a good or bad CDC algorithm, which were used by AE and FastCDC [11], [12]:…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

UltraCDC:A Fast and Stable Content-Defined Chunking Algorithm for Deduplication-based Backup Storage Systems

Zhou

Wang

Xia

et al. 2022

2022 IEEE International Performance, Computing, and Communications Conference (IPCCC)

Self Cite

View full text Add to dashboard Cite

show abstract

“…The earliest CDC algorithm is based on the Rabin fingerprint, 8 To pursue the chunking speed, Xia Wen et al proposed FastCDC. 9 In the fingerprint matching process, Gear fingerprint was used instead of Rabin, and two bytes were moved at a time to speed up the file traversal; at the same time, two kinds of fingerprint matching difficulties are used to increase the stability of the chunks. To reduce calculation in fingerprint matching, Bjrner Nikolaj et al proposed a new CDC algorithm based on the interval maximum value: LMC, 10 which determines to set the cut-point when the maximum value in a fixed window is in the middle of the window; the authors point out that LMC algorithm can be used to locate differential data in the application of file delta synchronization, and the feasibility of LMC is verified by theory and experiment.…”

Section: F I G U R E 1 Communication Flow Of Rsync Algorithmmentioning

confidence: 99%

SimpleSync: A parallel delta synchronization method based on Flink

Zhang

et al. 2021

Concurrency and Computation

View full text Add to dashboard Cite

Cloud storage service has been in full swing in the industry. Delta synchronization technology as a key technology of cloud storage services has not made a key breakthrough. Almost all the existing researches are based on the synchronization process proposed by the Rsync algorithm and mix some optimization appropriately, but the particularity of cloud storage service is not fully considered. This paper proposes a new incremental synchronization method SimpleSync, which makes use of the characteristic that the server does not actively modify the backup files in the cloud storage service, removes the redundant steps in Rsync, and enables the synchronization between the client and the server only through a single communication. Besides, according to the server-side synchronization request processing logic, this paper puts forward the design idea of parallel processing with the Flink framework, to the best of our knowledge, for the first time. After the server receives the synchronization request, SimpleSync first puts it into Kafka for buffering and then uses Flink to process the synchronization request in parallel. In the experimental part, a large number of experiments are designed to compare SimpleSync with other delta synchronization algorithms. Experimental results show that SimpleSync has obvious advantages in synchronization performance. Meanwhile, experiments show that SimpleSync has correctness.

show abstract

“…Almost 75% data in the data world are duplicated based on survey [1], and duplicated data reaches approximately 90% in the backup data and file system [2]. The development of deduplication technology in recent years has provided an effective solution for duplicated data, and deduplication is a data compression strategy applied to storage systems with high data compression rates [51][52][53][54][55][56].…”

Section: Introductionmentioning

confidence: 99%

Research on Data Routing Strategy of Deduplication in Cloud Environment

Zhang

Bian

et al. 2022

IEEE Access

View full text Add to dashboard Cite

The application of data deduplication technology reduces the demand for data storage and improves resource utilization. Compared with limited storage capacity and computing capacity of a single node, cluster data deduplication technology has great advantages. However, the cluster data duplication technology also brings new issues on deduplication rate reduction and load balancing of storage nodes. The application of data routing strategy can well balance the problem of deduplication rate and load balancing. Therefore, this paper proposes a data routing strategy based on distributed Bloom Filter. 1)Superchunk is used as the basic unit of data routing to improve system throughput. According to Broder's theorem, k leastsized fingerprints are selected as the Superchunk features and send to the storage node. The optimal node is selected as the routing node by matching the BloomFilter, and the storage capacity of the node and maintained in the memory of the storage node. 2) Design and implement system prototypes. The specific parameters of all kinds of routing strategies are obtained through experiments, and the routing strategies proposed in this paper are tested. The theoretical analysis and experimental results prove the feasibility of the strategies proposed by this paper. Compared with the other routing strategies, our method improved 3% of the deduplication rate, reduces the communication query overhead by more than 36% and improves the load balancing degree of the storage system.

show abstract

The Design of Fast Content-Defined Chunking for Data Deduplication Based Storage Systems

Cited by 53 publications

References 25 publications

UltraCDC:A Fast and Stable Content-Defined Chunking Algorithm for Deduplication-based Backup Storage Systems

UltraCDC:A Fast and Stable Content-Defined Chunking Algorithm for Deduplication-based Backup Storage Systems

SimpleSync: A parallel delta synchronization method based on Flink

Research on Data Routing Strategy of Deduplication in Cloud Environment

Contact Info

Product

Resources

About