The case for sampling on very large file systems

Abstract

Sampling has long been a prominent tool in statistics and analytics, first and foremost when very large amounts of data are involved. In the realm of very large file systems (and hierarchical data stores in general), however, sampling has mostly been ignored and for several good reasons. Mainly, running sampling in such an environment introduces technical challenges that make the entire sampling process non-beneficial. In this work we demonstrate that there are cases for which sampling is very worthwhile in very large file systems. We address this topic in two aspect: (a) the technical side where we design and implement solutions to efficient weighted sampling that is also distributed, one-pass and addresses multiple efficiency aspects; and (b) the usability aspect in which we demonstrate several use-cases in which weighted sampling over large file systems is extremely beneficial. In particular, we show use-cases regarding estimation of compression ratios, testing and auditing and offline collection of statistics on very large data stores.

Weighted Random Sampling over Data Streams

“…This algorithm, too, is efficient and can be applied to data streams. Random sampling is still an active research field and new sampling schemes are studied in various contexts; some indicative examples are sampling from sliding windows [13], from distributed data streams [4,15,5], from streams with time decay [6], independent range sampling [10], sampling on very large file systems [9], and stratified reservoir sampling [2]. In light of the above results (which are mainly from the data streams field), we consider the algorithms of [3] and [8] as fundamental sampling schemes for general purpose weighted random sampling over data streams.…”