Heavy hitters in streams and sliding windows

Ben-Basat, Ran; Einziger, Gil; Friedman, Roy; Kassner, Yaron

doi:10.1109/infocom.2016.7524364

Cited by 127 publications

(88 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Update Time Query Time Comments WCSS [8] O( −1 log(W |U |)) O(1) O(1) Only supports fixed-size window queries. ECM [33] O Table 1: Comparison of the algorithms proposed in the paper with ECM and WCSS (that solves the simpler problem of fixed-size windows).…”

Section: Spacementioning

confidence: 99%

“…While the traditional sliding window model can answer queries for a fixed window, our approach allows us to consider any interval that is contained within the last W items. In this example, we ask about the frequency of the item a within the interval [8,18]. If we allow an additive error of 2, the answer to this query should be in the range [4,6].…”

Section: Figurementioning

confidence: 99%

“…In [23], Hung and Ting improved the update time to O(1) while being able to find all heavy hitters in the optimal O( −1 ) time. Finally, the WCSS algorithm presented in [8] also estimates item frequencies in constant time. While some of these works also considered a variant in which the window can expand and shrink when processing updates [4,27], its size was increased/decreased by one at each update, and cannot be specified at query time.…”

Section: Related Workmentioning

confidence: 99%

“…That is, the sliding window model [18] considers only a window of the most recent items in the stream, while older ones do not affect the quantity we wish to estimate. Indeed, the problem of maintaining different types of sliding window statistics was extensively studied [4,8,18,33,27]. Yet, sometimes the window of interest may not be known a priori or they may be multiple interesting windows [17].…”

mentioning

confidence: 99%

See 3 more Smart Citations

Stream frequency over interval queries

Basat

Friedman²,

Shahout³

2018

Proc. VLDB Endow.

Self Cite

View full text Add to dashboard Cite

Stream frequency measurements are fundamental in many data stream applications such as financial data trackers, intrusion-detection systems, and network monitoring. Typically, recent data items are more relevant than old ones, a notion we can capture through a sliding window abstraction. This paper considers a generalized sliding window model that supports stream frequency queries over an interval given at query time. This enables drill-down queries, in which we can examine the behavior of the system in finer and finer granularities. For this model, we asymptotically improve the space bounds of existing work, reduce the update and query time to a constant, and provide deterministic solutions. When evaluated over real Internet packet traces, our fastest algorithm processes items 90-250 times faster, serves queries at least 730 times quicker and consumes at least 40% less space than the best known method. IntroductionHigh-performance stream processing is essential for many applications such as financial data trackers, intrusiondetection systems, network monitoring, and sensor networks. Such applications require algorithms that are both time and space efficient to cope with high-speed data streams. Space efficiency is needed, due to the memory hierarchy structure, to enable cache residency and to avoid page swapping. This residency is vital for obtaining good performance, even when the theoretical computational cost is small (e.g., constant time algorithms may be inefficient if they access the DRAM for each element). To that end, stream processing algorithms often build compact approximate sketches (synopses) of the input streams.Recent items are often more relevant than old ones, which requires an aging mechanism for the sketches. Many applications realize this by tracking the stream's items over a sliding window. That is, the sliding window model [18] considers only a window of the most recent items in the stream, while older ones do not affect the quantity we wish to estimate. Indeed, the problem of maintaining different types of sliding window statistics was extensively studied [4,8,18,33,27].Yet, sometimes the window of interest may not be known a priori or they may be multiple interesting windows [17]. Further, the ability to perform drill-down queries, in which we examine the behavior of the system in finer and finer granularity may also be beneficial, especially for security applications. For example, this enables detecting when precisely a particular anomaly has started and who was involved in it [20]. Additional applications for this capability include identifying the sources of flash crowd effects and pinpointing the cause-effect relation surrounding a surge in demand on an e-commerce website [26].In this work, we study a model that allows the user to specify an interval of interest at query time. This extends traditional sliding windows that only consider fixed sized windows. As depicted in Figure 1, a sub-interval of a maximal window is passed as a parameter for each query, and the goal of the algorithm...

show abstract

Section: Spacementioning

confidence: 99%

Section: Figurementioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Stream frequency over interval queries

Basat

Friedman²,

Shahout³

2018

Proc. VLDB Endow.

Self Cite

View full text Add to dashboard Cite

show abstract

“…Network measurement plays an important role in the evolution of large scale networks. Traffic statistics, such as top-K [24] and heavy hitters [3], are important for crucial network management applications. Cardinality estimation is the problem of estimating the number of distinct elements in a data stream with repeated elements.…”

Section: Introductionmentioning

confidence: 99%

Cardinality Estimation in a Virtualized Network Device Using Online Machine Learning

Cohen

Nezri

2019

IEEE/ACM Trans. Networking

View full text Add to dashboard Cite

Cardinality estimation algorithms receive a stream of elements, with possible repetitions, and return the number of distinct elements in the stream. Such algorithms seek to minimize the required memory and CPU resource consumption at the price of inaccuracy in their output. In computer networks, cardinality estimation algorithms are mainly used for counting the number of distinct flows, and they are divided into two categories: sketching algorithms and sampling algorithms. Sketching algorithms require the processing of all packets, and they are therefore usually implemented by dedicated hardware. Sampling algorithms do not require processing of all packets, but they are known for their inaccuracy. In this work we identify one of the major drawbacks of sampling-based cardinality estimation algorithms: their inability to adapt to changes in flow size distribution. To address this problem, we propose a new sampling-based adaptive cardinality estimation framework, which uses online machine learning. We evaluate our framework using real traffic traces, and show significantly better accuracy compared to the best known sampling-based algorithms, for the same fraction of processed packets.

show abstract

SA Sketch: A self‐adaption sketch framework for high‐speed network

Zhu

Yuan

Zhang

et al. 2020

Concurrency and Computation

View full text Add to dashboard Cite

Sketch is a compact data structure used to summarize data streams. It is widely used in the measurement of network traffic, and its accuracy is higher than traditional methods. Currently, there are some typical sketches: Count-Min Sketch, CU Sketch, and Count Sketch. According to the characteristics of network traffic, we propose a new sketch framework called Self-Adaption Sketch, which is combined Sketch with Bloom Filter. In the framework, the sketch is created dynamically and the memory space is adjusted timely according to the network traffic by using the concept carrying. Our experiment results showed that the space utilization and accuracy are significantly improved while the throughput of self-adaption sketch is maintained at a relatively good level. K E Y W O R D S traffic measurement, data stream technology, sketch, bloom filter, space utilization 1 INTRODUCTION 1.1 Background and motivation Network measurement and analysis are essential for network management and they help network administrators to understand the status of the network better and make the right decisions. They help to detect anomalies and attacks in the network. To provide a secure and well-performing network for the continually transforming cyberspace, Internet operators need to monitor and analyze the network status in real time. 1 In recent years, network traffic measurement has been widely used in network accounting, traffic engineering, network security, and other fields. However, there are certain difficulties in the measurement, which are mainly due to the continuous and massive network traffic. At present, there are two main types of high-speed network traffic measurement schemes, one is the sampling technology, and the other is the data stream technology. Sampling technology refers to the collection and processing of some representative network traffic. This method introduces the small system load with a relatively large error. Typical sampling methods are group sampling (such as BSS, 2 EBS 3) and stream sampling (such as ANF 4). Collapsed Gibbs sampling is the most commonly used algorithm that samples latent topics for a word occurrence (token) by integrating out the Dirichlet priors. 5 The sampling technique has been widely used, such as quantile calculation, data collection, and top-k query. 6 Data stream technology processes

show abstract

Heavy hitters in streams and sliding windows

Cited by 127 publications

References 19 publications

Stream frequency over interval queries

Stream frequency over interval queries

Cardinality Estimation in a Virtualized Network Device Using Online Machine Learning

SA Sketch: A self‐adaption sketch framework for high‐speed network

Contact Info

Product

Resources

About