Quantiles over data streams: experimental comparisons, new analyses, and further improvements

Abstract

A fundamental problem in data management and analysis is to generate descriptions of the distribution of data. It is most common to give such descriptions in terms of the cumulative distribution, which is characterized by the quantiles of the data. The design and engineering of efficient methods to find these quantiles has attracted much study, especially in the case where the data is given incrementally, and we must compute the quantiles in an online, streaming fashion. While such algorithms have proved to be extremely useful in practice, there has been limited formal comparison of the competing methods, and no comprehensive study of their performance. In this paper, we remedy this deficit by providing a taxonomy of different methods, and describe efficient implementations. In doing so, we propose new variants that have not been studied before, yet which outperform existing methods. To illustrate this, we provide detailed experimental comparisons demonstrating the tradeoffs between space, time, and accuracy for quantile computation.

Moment-based quantile sketches for efficient high cardinality aggregation queries

“…We compare against a number of alternative quantile summaries: a mergeable equi-width histogram (EW-Hist) using power-of-two ranges [65], the 'GKArray' (GK) variant of the Greenwald Khanna [34,52] sketch, the AVL-tree T-Digest (T-Digest) [28] sketch, the streaming histogram (S-Hist) in [12] as implemented in Druid, the 'Random' (RandomW) sketch from [52,77], reservoir sampling (Sampling) [76], and the low discrepancy mergeable sketch (Merge12) from [3], both implemented in the Yahoo! datasketches library [1].…”

Streaming Algorithms for Bin Packing and Vector Scheduling

“…Note that the lower and upper bounds on the rank of any stored number differ by at most 2δN and upper (or lower) bounds on the rank of two consecutive stored numbers differ by at most 2δN as well. The space requirement of Q(δ) is O( 1 δ · log δN ), however, in practice the space used is observed to scale linearly with 1 δ [36]. (Note that an offline optimal data structure for δ-approximate quantiles uses space O 1 δ .)…”

Untitled

“…Aside from oblivious sampling algorithms (which require storing Ω(1/ε 2 ) samples) the only other such work of which we are aware is an approach by Wang, Luo, Yi, and Cormode [12] that combines the methods of [1] and [8] into a hybrid with the same space bound as [1].…”

“…[1][2][3][4][5][6][7][8][9][10][11][12][13][14][15][16][17][18] For the comparison model, the best deterministic online summary to date is the (GK) summary of Greenwald and Khanna [4], which uses O((1/ε) log(εn)) space. This improved upon a deterministic (MRL) summary of Manku, Rajagopalan, and Lindsay [7] and a summary implied by Munro and Paterson [9], which use O((1/ε) log 2 (εn)) space.…”

“…This simple summary is not new (it is mentioned in Wang et al [12], for example) but the discussion provides exposition for Section 3, in which we develop this summary into a fully online summary with the same asymptotic space complexity that can answer queries at any point in time. At that point we will have proven the following theorem, which constitutes our main result.…”