Online maintenance of very large random samples

Abstract

Random sampling is one of the most fundamental data management tools available. However, most current research involving sampling considers the problem of how to use a sample, and not how to compute one. The implicit assumption is that a "sample" is a small data structure that is easily maintained as new data are encountered, even though simple statistical arguments demonstrate that very large samples of gigabytes or terabytes in size can be necessary to provide high accuracy. No existing work tackles the problem of maintaining very large, disk-based samples from a data management perspective, and no techniques now exist for maintaining very large samples in an online manner from streaming data. In this paper, we present online algorithms for maintaining on-disk samples that are gigabytes or terabytes in size. The algorithms are designed for streaming data, or for any environment where a large sample must be maintained online in a single pass through a data set. The algorithms meet the strict requirement that the sample always be a true, statistically random sample (without replacement) of all of the data processed thus far. Our algorithms are also suitable for biased or unequal probability sampling. IntroductionDespite the variety of alternatives for approximate query processing (including several references listed in this paper [8] [9] [10] [14][29]), sampling is still one of the most powerful methods for building a one-pass synopsis of a data set in a streaming environment, where the assumption is that there is too much data to store all of it permanently. Sampling's many benefits include:•Sampling is the most widely-studied and best understood approximation technique currently available. Sampling has been studied for hundreds of years, and many fundamental results describe the utility of random samples (such as the central limit theorem, Chernoff, Hoeffding and Chebyshev bounds [7][25]).•Sampling is the most versatile approximation technique available. Most data processing algorithms can be used on a random sample of a data set rather than the original data with little or no modification. For example, almost any data mining algorithm for building a decision tree classifier can be run directly on a sample.•Sampling is the most widely-used approximation technique. . However, this work is relevant mostly for sampling from data stored in a database, and is not suitable for emerging applications such as stream-based data management. Furthermore, the implicit assumption in most existing work is that a "sample" is a small, in-memory data structure. This is not always true. For many applications, very large samples containing billions of records can be required to provide acceptable accuracy. Fortunately, modern storage hardware gives us the capacity to cheaply store very large samples that should suffice for even difficult and emerging applications, such as futuristic "smart dust" environments where billions of tiny sensors produce billions of observations per second that must be joined, cross-correlated, a...

Online maintenance of very large random samples on flash storage

“…The fastest streaming algorithm for maintaining a large fixed-size random sample on a magnetic disk is due to Jermaine et al [11]. The algorithm uses an abstraction called the Geometric File.…”

“…In [11], the authors also propose using multiple geometric files in parallel for reducing the number of disk head movements. However, on flash, this scheme may not add a significant benefit since it does not reduce the amount of data overwrite and flash devices do not have any mechanical head movements.…”

“…In the basic version of the algorithm, a reservoir R is filled with the first n items (where n is the target size), and after that, the i'th item is selected for R with probability n/i. The selected item overwrites a random item in R. Many optimizations have been proposed to improve the performance of the basic algorithm [11,24]. Although the original algorithm is implicitly designed to maintain a sample in memory, it can be implemented on secondary storage.…”

“…However, like geometric file, they also require frequent in-place updates, making them unsuitable for flash. Moreover, as shown in [11], they require more random writes and hence perform worse than geometric file. Therefore, we do not consider them in the rest of the paper.…”

“…Our basic algorithm (see Algorithm 1) combines ideas from several sampling algorithms (e.g., [7,11]), in a novel way that is tailored to flash. When describing the algorithm, we will often highlight its adherence to the design principles (P1-P3) from Section 2.1.…”