Continuously maintaining quantile summaries of the most recent N elements over a data stream

Lin, Xuemin; Lü, Hongjun; Xu, Jie; Yu, Jeffrey Xu

doi:10.1109/icde.2004.1320011

Cited by 59 publications

(95 citation statements)

References 24 publications

(55 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In the sliding window model, where only the last W data elements are considered, the problem was first studied in [12] and improved in [1] to space O( 1 log 1 log W ).…”

Section: Related Workmentioning

confidence: 99%

Effective Computation of Biased Quantiles over Data Streams

Cormode¹,

Korn

Muthukrishnan

et al.

21st International Conference on Data Engineering (ICDE'05)

View full text Add to dashboard Cite

Skew is prevalent in many data sources such as IP traffic streams. To continually summarize the distribution of such data, a high-biased set of quantiles (e.g., 50th, 90th and 99th percentiles) with finer error guarantees at higher ranks (e.g., errors of 5, 1 and 0.1 percent, respectively) is more useful than uniformly distributed quantiles (e.g., 25th, 50th and 75th percentiles) can become very stretched. Hence, to gauge the performance of the network in detail and its effect on all users (not just those experiencing the average performance), it is important to know not only the median RTT but also the 90%, 95% and 99% quantiles of TCP round trip times to each destination. In developing data stream management systems that interact with IP traffic data, there exists the facility for posing such queries [4]. However, the challenge is to develop approaches to answer such queries efficiently and accurately given that there may be many destinations to track. In such settings, the data rate is typically high and resources are limited in comparison to the amount of data that is observed. Hence it is often necessary to adopt the data stream methodology [2,6,16]: analyze IP packet headers in one pass over the data with storage space and per-packet processing time that is significantly sublinear in the size of the input.Typically, IP traffic streams and other streams are summarized using quantiles: these are order statistics such as the minimum, maximum and median values. In a data set of size n, the φ-quantile is the item with rank φn .1 The minimum and maximum are easy to calculate precisely in one pass but exact computation of certain quantiles can require space linear in n [13]. So the notion of -approximate quantiles relaxes the requirement to finding an item with rank between (φ− )n and (φ+ )n. Much attention has been given to the case of finding a set of uniform quantiles: given 0 < φ < 1, return the approximate φ, 2φ, 3φ, . . . , 1/φ φ quantiles of a stream of values.2 Note that the error in the rank of each returned value is bounded by the same amount, n; we call this the uniform error case.Summarizing distributions which have high skew using uniform quantiles is not always informative because it 1 We use the rank of an item to refer to its position in the sorted order of items that have been observed.2 While existing formal problem definitions (eg, [7]) find a single order statistic at rank φn , it is trivial to modify the output routine to return uniform quantiles as defined above; this is consistent with how "quantile" is defined in the statistics literature (eg, "percentiles" when φ = 0.01).

show abstract

“…In the sliding window model, where only the last W data elements are considered, the problem was first studied in [12] and improved in [1] to space O( 1 log 1 log W ).…”

Section: Related Workmentioning

confidence: 99%

Effective Computation of Biased Quantiles over Data Streams

Cormode¹,

Korn

Muthukrishnan

et al.

21st International Conference on Data Engineering (ICDE'05)

View full text Add to dashboard Cite

show abstract

“…Both deterministic and randomized methods are proposed. In our implementation, we adopt the method of computing approximate quantile summaries in a sliding window proposed in [41], which is based on the GK-algorithm [26] that finds the approximate quantile over a data steam. The algorithm can continuously output the -approximate quantiles in a sliding window with space cost of O( log 2 ω 2 ).…”

Section: Definition 11 ( -Approximate Quantile) Let O 1 ≺ · · · ≺ O ωmentioning

confidence: 99%

“…Computing -Approximate quantiles in data streams is well studied [25,26,41,43]. Both deterministic and randomized methods are proposed.…”

Section: Definition 11 ( -Approximate Quantile) Let O 1 ≺ · · · ≺ O ωmentioning

confidence: 99%

Continuously monitoring top-k uncertain data streams: a probabilistic threshold method

Hua

Pei

2009

Distrib Parallel Databases

View full text Add to dashboard Cite

Recently, uncertain data processing has become more and more important. Although a significant amount of previous research explores various continuous queries on data streams, continuous queries on uncertain data streams have seldom been investigated. In this paper, we formulate a novel and challenging problem of continuously monitoring top-k uncertain data streams, and propose a probabilistic threshold method. We develop four algorithms systematically: a deterministic exact algorithm, a randomized method, and their space-efficient versions using quantile summaries. An extensive empirical study using real data sets and synthetic data sets is reported to verify the effectiveness and the efficiency of our methods.

show abstract

“…The algorithm uses O(log 2 |U | log (log(|U |)/δ)/ 2 ) space, given the value domain U of data and the algorithm estimates -approximate quantiles with confidence 1 − δ. The only work we know for estimating quantiles on sliding windows is [6]. In this paper, the authors gave a solution for estimating quantiles on a sliding window containing a fixed number of data items.…”

Section: Introductionmentioning

confidence: 99%

“…Firstly, the number of data items in the window is unknown and changes over time. The technique in [6] does not work without an apriori to the window size. Secondly, constraints may have various of forms and a practical query processing should be able to handle different window queries in a uniform way rather than hacking from one specific solution to another.…”

Section: Introductionmentioning

confidence: 99%

Space Efficient Quantile Summary for Constrained Sliding Windows on a Data Stream

Lin

Zhou

2004

Advances in Web-Age Information Management

View full text Add to dashboard Cite

Abstract. In many online applications, we need to maintain quantile statistics for a sliding window on a data stream. The sliding windows in natural form are defined as the most recent N data items. In this paper, we study the problem of estimating quantiles over other types of sliding windows. We present a uniform framework to process quantile queries for time constrained and filter based sliding windows. Our algorithm makes one pass on the data stream and maintains an -approximate summary. It uses O( 1 2 log 2 N ) space where N is the number of data items in the window. We extend this framework to further process generalized constrained sliding window queries and proved that our technique is applicable for flexible window settings. Our performance study indicates that the space required in practice is much less than the given theoretical bound and the algorithm supports high speed data streams.

show abstract

Continuously maintaining quantile summaries of the most recent N elements over a data stream

Cited by 59 publications

References 24 publications

Effective Computation of Biased Quantiles over Data Streams

Effective Computation of Biased Quantiles over Data Streams

Continuously monitoring top-k uncertain data streams: a probabilistic threshold method

Space Efficient Quantile Summary for Constrained Sliding Windows on a Data Stream

Contact Info

Product

Resources

About