Hadoop Based Scalable Cluster Deduplication for Big Data

The enormous growth of digital data, especially the data in unstructured format has brought a tremendous challenge on data analysis as well as the data storage systems which are essentially increasing the cost and performance of the backup systems. The traditional systems do not provide any optimization techniques to keep the duplicated data from being backed up. Deduplication of data has become an essential and financial way of the capacity optimization technique which replaces the redundant data. The following paper reviews the deduplication process, types of deduplication and techniques available for data deduplication. Also, many approaches proposed by various researchers on deduplication in Big data storage systems are studied and compared.

show abstract

“…Deduplication [10] Unique data storage is done using HDFS; Map Reduce is used to realize parallel deduplication processing.…”

Section: Chunk Levelmentioning

confidence: 99%

Data Deduplication Techniques for Big Data Storage Systems

Sharma¹,

Prasad²,

Kakulapati³

2019

IJITEE

View full text Add to dashboard Cite

show abstract

“…Data deduplication technology generally divides files into chunks and calculates the fingerprint of each data chunk by a hash function. The fingerprints are compared to detect whether two data chunks are redundant or not [6]. Dedu-plication technology keeps only one duplicate to reduce the amount of data to save storage space during the data storage process [7], [8].…”

Section: Introductionmentioning

confidence: 99%

FASR: An Efficient Feature-Aware Deduplication Method in Distributed Storage Systems

et al. 2022

View full text Add to dashboard Cite

Deduplication technology can obtain higher space utilization by keeping only one duplicate. But in a distributed storage system, the overall deduplication ratio will be limited due to redundancy elimination across nodes. The traditional deduplication methods usually utilize data similarity and data locality to improve the deduplication ratio. However, higher system overhead is caused by frequent similarity calculations. To deal with this problem, this paper proposes a new Feature-Aware Stateful Routing method(FASR), aiming to reduce the system overhead and keep a high deduplication ratio in the distributed environment. Firstly, we design a feature-aware nodes selection strategy to choose similar nodes by extracting data feature and data distribution characteristics. This strategy will save the similarity calculation with the nodes that are not similar to the data. Then, we present a stateful routing algorithm to determine the target node using super-chunk and handprint technology. Meanwhile, the algorithm maintain load balance of the entire distributed system. Finally, the data is deduplicated locally based on similarity index and fingerprint cache. Extensive experiments demonstrate that FASR can reduce system overhead by around 30% at most and also effectively obtain a higher deduplication ratio.

show abstract