Conditional heavy hitters: detecting interesting correlations in data streams

Abstract

The notion of heavy hitters-items that make up a large fraction of the population-has been successfully used in a variety of applications across sensor and RFID monitoring, network data analysis, event mining, and more. Yet this notion often fails to capture the semantics we desire when we observe data in the form of correlated pairs. Here, we are interested in items that are conditionally frequent: when a particular item is frequent within the context of its parent item. In this work, we introduce and formalize the notion of Conditional Heavy Hitters to identify such items, with applications in network monitoring, and Markov chain modeling. We explore the relationship between Conditional Heavy Hitters and other related notions in the literature, and show analytically and experimentally the usefulness of our approach. We introduce several algorithm variations that allow us to efficiently find conditional heavy hitters for input data with very different characteristics, and provide analytical results for their performance. Finally, we perform experimental evaluations with several synthetic and real datasets to demonstrate the efficacy of our methods, and to study the behavior of the proposed algorithms for different types of data.

Sketching Linear Classifiers over Data Streams

“…Algorithms for finding frequent items include counter-based approaches (Manku and Motwani, 2002;Demaine et al, 2002;Karp et al, 2003;Metwally et al, 2005), quantile algorithms (Greenwald and Khanna, 2001;Shrivastava et al, 2004), and sketch-based methods (Charikar et al, 2002;Cormode and Muthukrishnan, 2005b). Mirylenka et al (Mirylenka et al, 2015) develop streaming algorithms for finding conditional heavy hitters, i.e. items that are frequent in the context of a separate "parent" item.…”

Fast and accurate mining of correlated heavy hitters

“…In [25], Mirylenka et al introduce the notion of Conditional Heavy Hitters and compare it with other related problems, such as association rules and Correlated Heavy Hitters, highlighting how solving these problems actually leads to different outputs, each emphasizing particular aspects of the input data stream. A group of algorithms is proposed and experimentally evaluated with respect to the approximate mining of Conditional Heavy Hitters.…”

HeavyKeeper: An Accurate Algorithm for Finding Top-$k$ Elephant Flows

“…Many management applications can benefit from a function that can find them efficiently, such as congestion control by dynamically scheduling elephant flows [2], network capacity planning [3], anomaly detection [4], and caching of forwarding table entries [5]. Such a function not only is important in networking measurements [6]- [15], but also has applications beyond networking in areas such as data mining [16]- [18], information retrieval [19], databases [20], and security [21].…”