Lazy exact deduplication

Ma, Jingwei; Stones, Rebecca J.; Ma, Yuxiang; Wang, Jingui; Ren, Junjie; Wang, Gang; Liu, Xiaoguang

doi:10.1109/msst.2016.7897081

Cited by 7 publications

(2 citation statements)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Kaiser et al [47] present an inline deduplication cluster with a jointly distributed chunk index, which is able to detect as many data redundancies as a single node solution. Literature [48] proposes a lazy data deduplication method. The method buffers incoming fingerprints that are used to perform on-disk lookups in batches, with the aim of improving subsequent prefetching.…”

Section: Related Workmentioning

confidence: 99%

LOFS: A Lightweight Online File Storage Strategy for Effective Data Deduplication at Network Edge

Cheng

Guo

Xia

et al. 2022

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

Edge computing responds to users' requests with low latency by storing the relevant files at the network edge. Various data deduplication technologies are currently employed at edge to eliminate redundant data chunks for space saving. However, the lookup for the global huge-volume fingerprint indexes imposed by detecting redundancies can significantly degrade the data processing performance. Besides, we envision a novel file storage strategy that realizes the following rationales simultaneously: 1) space efficiency, 2) access efficiency, and 3) load balance, while the existing methods fail to achieve them at one shot. To this end, we report LOFS, a Lightweight Online File Storage strategy, which aims at eliminating redundancies through maximizing the probability of successful data deduplication, while realizing the three design rationales simultaneously. LOFS leverages a lightweight three-layer hash mapping scheme to solve this problem with constant-time complexity. To be specific, LOFS employs the Bloom filter to generate a sketch for each file, and thereafter feeds the sketches to the Locality Sensitivity hash (LSH) such that similar files are likely to be projected nearby in LSH tablespace. At last, LOFS assigns the files to real-world edge servers with the joint consideration of the LSH load distribution and the edge server capacity. Trace-driven experiments show that LOFS closely tracks the global deduplication ratio and generates a relatively low load std compared with the comparison methods.

show abstract

Section: Related Workmentioning

confidence: 99%

LOFS: A Lightweight Online File Storage Strategy for Effective Data Deduplication at Network Edge

Cheng

Guo

Xia

et al. 2022

IEEE Trans. Parallel Distrib. Syst.

View full text Add to dashboard Cite

show abstract

“…DDFS has been used in production backup management for over 15 years [5,23]. Its chunk locality design also lays the foundation of various follow-up deduplicated storage systems (e.g., [45,47,65]). Thus, we believe that DDFS is representative and our study results for DDFS can also be applied to other locality-based deduplicated storage systems.…”

Section: Metadata Access Overheadmentioning

confidence: 99%

Information Leakage in Encrypted Deduplication via Frequency Analysis

Qin

Lee

et al. 2017

2017 47th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN)

View full text Add to dashboard Cite

Encrypted deduplication combines encryption and deduplication to simultaneously achieve both data security and storage efficiency. State-of-the-art encrypted deduplication systems mainly build on deterministic encryption to preserve deduplication effectiveness. However, such deterministic encryption reveals the underlying frequency distribution of the original plaintext chunks. This allows an adversary to launch frequency analysis against the ciphertext chunks and infer the content of the original plaintext chunks. In this paper, we study how frequency analysis affects information leakage in encrypted deduplication, from both attack and defense perspectives. Specifically, we target backup workloads, and propose a new inference attack that exploits chunk locality to increase the coverage of inferred chunks. We further combine the new inference attack with the knowledge of chunk sizes and show its attack effectiveness against variable-size chunks. We conduct trace-driven evaluation on both real-world and synthetic datasets and show that our proposed attacks infer a significant fraction of plaintext chunks under backup workloads. To defend against frequency analysis, we present two defense approaches, namely MinHash encryption and scrambling. Our trace-driven evaluation shows that our combined MinHash encryption and scrambling scheme effectively mitigates the severity of the inference attacks, while maintaining high storage efficiency and incurring limited metadata access overhead.

show abstract