Deferred Maintenance of Disk-Based Random Samples

External Memory Stream Sampling

“…Therefore, the algorithm of [7], as well as its WR extension described in Section 1.3, is already optimal. Note that the theorem officially separates WoR sampling from WR sampling-conditioned on spending O( R B log N R ) I/Os processing N stream elements, a WoR sample set can be reported in O(R/B) I/Os, whereas a WR one requires Ω(perm(R)) I/Os to report.…”

“…The geometric file of Jermaine et al [12,18] processes a stream of N elements with total cost of O( R 2 M B log N R ) expected I/Os, and consumes O(R/B) space at all times, such that a sample set of size R can be output at any moment in O(R/B) I/Os. Gemulla and Lehner [7] presented an improved algorithm that (when adapted to EMS) has the same space and sample reporting cost as the geometric file, but can process N stream elements with O( R B log N R ) expected I/Os in total. There have been no explicit studies on maintaining massive WR sample sets.…”

“…However, the folklore RAM algorithm for size-R WoR-to-WR conversion can be implemented in perm(R) = Θ(min{R, R B log M/B R B }) I/Os in EM. 2 Hence, the algorithm of [7] can also output a size-R WR sample set in perm(R) I/Os. An interesting question then arises: is it possible to carry out a size-R WoR-to-WR conversion in o(perm(R)) I/Os?…”

“…Request permissions from permissions@acm.org. triggered a line of research [7,12,15,16,18] on this topic. The importance of big sampling is reflected in the fact that, the sample size must grow rapidly with the cardinality of the "ground" input set to allow high accuracy in many analytic tasks.…”

“…In this subsection, we review the algorithm of [7] for solving Problem 1, which can be regarded as an implementation of reservoir sampling in the EMS model.…”