Randomized algorithms for tracking distributed count, frequencies, and ranks

Huang, Zengfeng; Yi, Ke; Zhang, Qin

doi:10.1145/2213556.2213596

Cited by 42 publications

(62 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Similarly, Huang et al [26] explore a filtering-based approach that leverages in-network processing to detect (PCA-based) network anomalies. More recent theoretical work using the continuous streaming model [19], [20], [28], [29], [38] proposes communication efficient algorithms and gives bounds to calculate aggregation functions approximately. In contrast to the continuous streaming model, our adaptive aggregation algorithm works in batches of consecutive windows where the goal is to detect global icebergs in the recent past (say the last 1 minute).…”

Section: Related Workmentioning

confidence: 99%

Pegasus: Precision hunting for icebergs and anomalies in network flows

Gangam

Sharma²,

Fahmy

2013

2013 Proceedings IEEE INFOCOM

View full text Add to dashboard Cite

Abstract-Accurate online network monitoring is crucial for detecting attacks, faults, and anomalies, and determining traffic properties across the network. With high bandwidth links and consequently increasing traffic volumes, it is difficult to collect and analyze detailed flow records in an online manner. Traditional solutions that decouple data collection from analysis resort to sampling and sketching to handle large monitoring traffic volumes. We propose a new system, Pegasus, to leverage commercially available co-located compute and storage devices near routers and switches. Pegasus adaptively manages data transfers between monitors and aggregators based on traffic patterns and user queries.We use Pegasus to detect global icebergs or global heavy-hitters. Icebergs are flows with a common property that contribute a significant fraction of network traffic. For example, DDoS attack detection is an iceberg detection problem with a common destination IP. Other applications include identification of "top talkers," top destinations, and detection of worms and port scans. Experiments with Abilene traces, sFlow traces from an enterprise network, and deployment of Pegasus as a live monitoring service on PlanetLab show that our system is accurate and scales well with increasing traffic and number of monitors.

show abstract

Section: Related Workmentioning

confidence: 99%

Pegasus: Precision hunting for icebergs and anomalies in network flows

Gangam

Sharma²,

Fahmy

2013

2013 Proceedings IEEE INFOCOM

View full text Add to dashboard Cite

show abstract

“…Cormode et al [39] pioneer the formal study of functions in this model by focusing on the estimation of the first three frequency moments F 0 , F 1 and F 2 [33]. Arackaparambil et al [40] consider the empirical entropy estimation [33] and improve the work of Cormode by providing lower bounds on the frequency moments, and finally distributed algorithms for counting at any time t the number of items that have been received by a set of nodes from the inception of their streams have been proposed in [42], [43]. However, an open research issue is:…”

Section: Rq 4-how Can Group Communication Paradigms Contribute To Permentioning

confidence: 99%

Reinventing Mobile Community Computing and Communication

Busnel

Cruz

Gillet

et al. 2013

2013 12th IEEE International Conference on Trust, Security and Privacy in Computing and Communications

View full text Add to dashboard Cite

8 pagesInternational audienceIn less than a decade, smartphones and mobile applications spread like wildfire and dramatically improved aspects of our professional and private lives, from efficiency to safety. However, these applications are still in their infancy and mostly provide mobile versions of online Internet services or arcade games. With the exception of simple location-based query applications, context-awareness is largely ignored. However, it is not hard to imagine advanced mobile social networking applications - SNAPPs for short - that could proactively assist users in everyday tasks, improving their quality of life. Such services would require massive data collection, processing and communication between mobile devices. Unfortunately, the current centralised communication paradigm represents a major barrier to such intense networking. In this paper, we claim that a fundamental paradigm shift in communication is required to allow such application to see the light of day. The paper claims that such a shift is possible and that it resides in moving towards decentralised communication by taking advantage of the largely untapped network, storage and processing power capabilities offered by idle mobile devices. The paper presents and discusses a number of research questions that must be addressed in order to achieve this paradigm shif

show abstract

“…It follows that any communication complexity lower bound in the messagepassing model or the coordinator model also hold in the distributed monitoring model. A lot of work on distributed monitoring has been done recently in the theory community and the database community, including maintaining random samplings [15], frequency moments [12,14], heavy hitters [5,27,30,43,24], quantiles [13,43], entropy [4], various sketches [16,13] and some non-linear functions [37,38].…”

Section: Motivation Previous Work and Related Modelsmentioning

confidence: 99%

Lower Bounds for Number-in-Hand Multiparty Communication Complexity, Made Easy

Phillips¹,

Verbin²,

Zhang³

2012

Proceedings of the Twenty-Third Annual ACM-SIAM Symposium on Discrete Algorithms

Self Cite

View full text Add to dashboard Cite

In this paper we prove lower bounds on randomized multiparty communication complexity, both in the blackboard model (where each message is written on a blackboard for all players to see) and (mainly) in the message-passing model, where messages are sent player-to-player. We introduce a new technique for proving such bounds, called symmetrization, which is natural, intuitive, and often easy to use.For example, for the problem where each of k players gets a bit-vector of length n, and the goal is to compute the coordinate-wise XOR of these vectors, we prove a tight lower bounds of Ω(nk) in the blackboard model. For the same problem with AND instead of XOR, we prove a lower bounds of roughly Ω(nk) in the message-passing model (assuming k ≤ n/3200) and Ω(n log k) in the blackboard model. We also prove lower bounds for bit-wise majority, for a graph-connectivity problem, and for other problems; the technique seems applicable to a wide range of other problems as well. All of our lower bounds allow randomized communication protocols with two-sided error.We also use the symmetrization technique to prove several direct-sum-like results for multiparty communication.

show abstract

Randomized algorithms for tracking distributed count, frequencies, and ranks

Cited by 42 publications

References 33 publications

Pegasus: Precision hunting for icebergs and anomalies in network flows

Pegasus: Precision hunting for icebergs and anomalies in network flows

Reinventing Mobile Community Computing and Communication

Lower Bounds for Number-in-Hand Multiparty Communication Complexity, Made Easy

Contact Info

Product

Resources

About