Quick Estimation of Data Compression and De-duplication for Large Storage Systems

Constantinescu, Corneliu; Lu, Maohua

doi:10.1109/ccp.2011.41

Cited by 9 publications

(5 citation statements)

References 3 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[6], [24]) has shown that in many large scale file systems, small files account for a large number of the files but only a small portion of total capacity (in some cases 99% of the files accounted for less then 10% of the capacity). So sampling randomly from the entire list of files may yield a sample set with a disproportional number of small files that actually have very little effect on the overall capacity.…”

Section: Estimation Via Sampling -Preliminariesmentioning

confidence: 98%

See 1 more Smart Citation

The case for sampling on very large file systems

Goldberg

Harnik

Sotnikov

2014

2014 30th Symposium on Mass Storage Systems and Technologies (MSST)

View full text Add to dashboard Cite

Sampling has long been a prominent tool in statistics and analytics, first and foremost when very large amounts of data are involved. In the realm of very large file systems (and hierarchical data stores in general), however, sampling has mostly been ignored and for several good reasons. Mainly, running sampling in such an environment introduces technical challenges that make the entire sampling process non-beneficial. In this work we demonstrate that there are cases for which sampling is very worthwhile in very large file systems. We address this topic in two aspect: (a) the technical side where we design and implement solutions to efficient weighted sampling that is also distributed, one-pass and addresses multiple efficiency aspects; and (b) the usability aspect in which we demonstrate several use-cases in which weighted sampling over large file systems is extremely beneficial. In particular, we show use-cases regarding estimation of compression ratios, testing and auditing and offline collection of statistics on very large data stores.

show abstract

Section: Estimation Via Sampling -Preliminariesmentioning

confidence: 98%

“…Finally, compression estimation was studied in [6] and [11]. The first paper sampled parts of each file which amounts to worst performance than our method and without accuracy guarantees.…”

Section: Related Workmentioning

confidence: 99%

The case for sampling on very large file systems

Goldberg

Harnik

Sotnikov

2014

2014 30th Symposium on Mass Storage Systems and Technologies (MSST)

View full text Add to dashboard Cite

show abstract

“…Estimation Accuracy: We compare our CLA size estimators with a systematic excerpt [17] (first 0.01n rows), that allows to observe compression ratios. Table 5 reports the ARE (absolute ratio error) |Ŝ − S|/S of estimated sizeŜ (before corrections) to actual CLA compressed size S. CLA shows significantly better accuracy due to robustness against skew and effects of value tuples.…”

Section: Parameter Influence and Accuracymentioning

confidence: 99%

“…Compression Planning: The literature for compression and deduplication planning is relatively sparse and focuses on a priori estimation of compression ratios for heavyweight algorithms on generic data. A common strategy [17] is to experimentally compress a small segment of the data (excerpt) and observe the compression ratio. The drawbacks to this approach [26] are that (1) the segment may not be representative of the whole dataset and (2) the compression step can be very expensive because the runtime of many algorithms varies inversely with the achieved compression.…”

Section: Related Workmentioning

confidence: 99%

Compressed linear algebra for large-scale machine learning

et al. 2016

View full text Add to dashboard Cite

Large-scale machine learning (ML) algorithms are often iterative, using repeated read-only data access and I/Obound matrix-vector multiplications to converge to an optimal model. It is crucial for performance to fit the data into single-node or distributed main memory. General-purpose, heavy-and lightweight compression techniques struggle to achieve both good compression ratios and fast decompression speed to enable block-wise uncompressed operations. Hence, we initiate work on compressed linear algebra (CLA), in which lightweight database compression techniques are applied to matrices and then linear algebra operations such as matrix-vector multiplication are executed directly on the compressed representations. We contribute effective column compression schemes, cache-conscious operations, and an efficient sampling-based compression algorithm. Our experiments show that CLA achieves in-memory operations performance close to the uncompressed case and good compression ratios that allow us to fit larger datasets into available memory. We thereby obtain significant end-to-end performance improvements up to 26x or reduced memory requirements.

show abstract

“…For example, to get the accurate size of a subset of files, one needs to deduplicate the subset of files with the help of the deduplication metadata. A quick method to estimate the deduplicated size of a file system before performing deduplication, thus without knowing the full deduplication metadata, is described by Constantinescu et al [22].…”

Section: Management Of Files On Deduplication-enabled Storagementioning

confidence: 99%

Content Sharing Graphs for Deduplication-Enabled Storage Systems

Constantinescu

Sarkar

2012

Algorithms

Self Cite

View full text Add to dashboard Cite

Deduplication in storage systems has gained momentum recently for its capability in reducing data footprint. However, deduplication introduces challenges to storage management as storage objects (e.g., files) are no longer independent from each other due to content sharing between these storage objects. In this paper, we present a graph-based framework to address the challenges of storage management due to deduplication. Specifically, we model content sharing among storage objects by content sharing graphs (CSG), and apply graph-based algorithms to two real-world storage management use cases for deduplication-enabled storage systems. First, a quasi-linear algorithm was developed to partition deduplication domains with a minimal amount of deduplication loss (i.e., data replicated across partitioned domains) in commercial deduplication-enabled storage systems, whereas in general the partitioning problem is NP-complete. For a real-world trace of 3 TB data with 978 GB of removable duplicates, the proposed algorithm can partition the data into 15 balanced partitions with only 54 GB of deduplication loss, that is, a 5% deduplication loss. Second, a quick and accurate method to query the deduplicated size for a subset of objects in deduplicated storage systems was developed. For the same trace of 3 TB data, the optimized graph-based algorithm can complete the query in 2.6 s, which is less than 1% of that of the traditional algorithm based on the deduplication metadata

show abstract

Quick Estimation of Data Compression and De-duplication for Large Storage Systems

Cited by 9 publications

References 3 publications

The case for sampling on very large file systems

The case for sampling on very large file systems

Compressed linear algebra for large-scale machine learning

Content Sharing Graphs for Deduplication-Enabled Storage Systems

Contact Info

Product

Resources

About