Leveraging Data Deduplication to Improve the Performance of Primary Storage Systems in the Cloud

Abstract

With the explosive growth in data volume, the I/O bottleneck has become an increasingly daunting challenge for big data analytics in the Cloud. Recent studies have shown that moderate to high data redundancy clearly exists in primary storage systems in the Cloud. Our experimental studies reveal that data redundancy exhibits a much higher level of intensity on the I/O path than that on disks due to relatively high temporal access locality associated with small I/O requests to redundant data. Moreover, directly applying data deduplication to primary storage systems in the Cloud will likely cause space contention in memory and data fragmentation on disks. Based on these observations, we propose a performance-oriented I/O deduplication, called POD, rather than a capacityoriented I/O deduplication, exemplified by iDedup, to improve the I/O performance of primary storage systems in the Cloud without sacrificing capacity savings of the latter. POD takes a two-pronged approach to improving the performance of primary storage systems and minimizing performance overhead of deduplication, namely, a request-based selective deduplication technique, called Select-Dedupe, to alleviate the data fragmentation and an adaptive memory management scheme, called iCache, to ease the memory contention between the bursty read traffic and the bursty write traffic. We have implemented a prototype of POD as a module in the Linux operating system. The experiments conducted on our lightweight prototype implementation of POD show that POD significantly outperforms iDedup in the I/O performance measure by up to 87.9% with an average of 58.8%. Moreover, our evaluation results also show that POD achieves comparable or better capacity savings than iDedup.

Cdep: Qos-Aware Crowd-Deduplication with Efficient Data Placement in Big Data Analytics

“…Many of us lead often the impersonating lab tests using staying weights coming from three stockpiling structures on the Computer Research office, which includes two net laborers (web-vm), an email expert (mail) along with a store laborer [6]. The I/O follows have been assembled downstream of a functioning page free from each and every structure to get a term regarding three days.…”

“…Progressing International Data Corporation (IDC) examines demonstrate that lately the quantity of data has expanded by very nearly multiple times to 7ZB consistently and a more than 44overlap development to 35ZB is typical in the accompanying ten years [4]. Dealing with the data storm on storage to help (close) consistent data examination turns into an undeniably basic test for Big Data investigation in the Server, particularly for VM stages where the absolute number and strength of little documents overpower the I/O information route in the Server [6].…”

Off-line Deduplication Method for Solid-State Disk Based on Hot and Cold Data

“…However, the size of the mapping table increased due to the introduction of the secondary mapping table. A. Gupta et al [13,14] used the value locality inside SSDs to reduce their write load while realizing deduplication. However, evaluating the efficiency of deduplication is impossible because they did not discuss fingerprint generation.…”

Systematic Literature Review: Leveraging Data Deduplication Strategies & Hashing Techniques to Eliminate Data Redundancy in Cloud Environments

“…Numerous researchers depicted 'Data Deduplication' is the most efficient strategy to resolve the multiple data copies in storage [6] and [7].Data deduplication (dedup) techniques are extensively used to eliminate duplicate copy of data in cloud storage [8], [9]. Rather than only reducing the storage overheads, deduplication similarly capable to optimized the bandwidth usage [10].…”