Functional Monitoring without Monotonicity

Arackaparambil, Chrisil; Brody, Joshua; Chakrabarti, Amit

doi:10.1007/978-3-642-02927-1_10

Cited by 67 publications

(98 citation statements)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…All protocols in the distributed streaming model are also valid protocols in our one-shot computational model, while our impossibility results in our one-shot computational model also apply to all protocols in the distributed streaming model. Example functions studied in the distributed streaming model include F 0 [7], F 2 (size of self join) [7,27], quantile and heavy-hitters [16], and the empirical entropy [3]. All of these problems have much lower communication cost if one allows an approximation of the output number x in a range [(1 − ε)x, (1 + ε)x], as mentioned above (the definition as to what ε is for the various problems differs).…”

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

When Distributed Computation Is Communication Expensive

Woodruff

Zhang

2013

Lecture Notes in Computer Science

View full text Add to dashboard Cite

We consider a number of fundamental statistical and graph problems in the message-passing model, where we have k machines (sites), each holding a piece of data, and the machines want to jointly solve a problem defined on the union of the k data sets. The communication is point-to-point, and the goal is to minimize the total communication among the k machines. This model captures all point-to-point distributed computational models with respect to minimizing communication costs. Our analysis shows that exact computation of many statistical and graph problems in this distributed setting requires a prohibitively large amount of communication, and often one cannot improve upon the communication of the simple protocol in which all machines send their data to a centralized server. Thus, in order to obtain protocols that are communication-efficient, one has to allow approximation, or investigate the distribution or layout of the data sets.

show abstract

Section: Related Workmentioning

confidence: 99%

“…This is because in the simulation he locally generates all X i (i ∈ [k]\I) together with Y . On the other hand, by a union bound, the probability that P is correct for all three input reductions is at least 1 − 3/k 3 . Note that if we can compute f k OR (X 1 , .…”

Section: Theorem 1 For Any Functionmentioning

confidence: 99%

When Distributed Computation Is Communication Expensive

Woodruff

Zhang

2013

Lecture Notes in Computer Science

View full text Add to dashboard Cite

show abstract

“…In the communication complexity model, each node receives an input data stream, performs some local computation, and communicates only with a coordinator who wishes to continuously compute or estimate a given function of the union of all the input streams. The challenging issue in this model is for the coordinator to compute the given function by minimizing the number of communicated bits [17], [18], [19]. Cormode et al [17] pioneer the formal study of functions in this model by focusing on the estimation of the first three frequency moments F 0 , F 1 and F 2 [11].…”

Section: Introductionmentioning

confidence: 99%

“…Cormode et al [17] pioneer the formal study of functions in this model by focusing on the estimation of the first three frequency moments F 0 , F 1 and F 2 [11]. Arackaparambil et al [18] consider the empirical entropy estimation [11] and improve the work of Cormode by providing lower bounds on the frequency moments, and finally distributed algorithms for counting at any time t the number of items that have been received by a set of nodes from the inception of their streams have been proposed in [20], [21].…”

Section: Introductionmentioning

confidence: 99%

Deviation Estimation between Distributed Data Streams

Anceaume

Busnel

2014

2014 Tenth European Dependable Computing Conference

View full text Add to dashboard Cite

Abstract-The analysis of massive data streams is fundamental in many monitoring applications. In particular, for networks operators, it is a recurrent and crucial issue to determine whether huge data streams, received at their monitored devices, are correlated or not as it may reveal the presence of malicious activities in the network system. We propose a metric, called codeviation, that allows to evaluate the correlation between distributed streams. This metric is inspired from classical metric in statistics and probability theory, and as such allows us to understand how observed quantities change together, and in which proportion. We then propose to estimate the codeviation in the data stream model. In this model, functions are estimated on a huge sequence of data items, in an online fashion, and with a very small amount of memory with respect to both the size of the input stream and the values domain from which data items are drawn. We give upper and lower bounds on the quality of the codeviation, and provide both local and distributed algorithms that additively approximates the codeviation among n data streams by using O ((1/ε) log(1/δ) (log N + log m)) bits of space for each of the n nodes, where N is the domain value from which data items are drawn, and m is the maximal stream's length. To the best of our knowledge, such a metric has never been proposed so far.

show abstract

“…Thus, one has to consider randomized approximation algorithms to have any hope of computing entropy efficiently. This problem of monitoring entropy efficiently has received widespread interest in the theory community, and several data streaming algorithms are now known [2], [3], [6], [7], [11], [12], [16] that utilize significantly lower memory than straightforward computation. However, even such algorithms fail to keep up with everincreasing channel bandwidths observed at a typical NOC of a large organization, especially considering that monitoring of several stream features is desirable for accurate detection.…”

Section: Introductionmentioning

confidence: 99%

Distributed monitoring of conditional entropy for anomaly detection in streams

Arackaparambil

Bratus

Brody

et al. 2010

2010 IEEE International Symposium on Parallel &Amp; Distributed Processing, Workshops and PHD Forum (IPDPSW)

Self Cite

View full text Add to dashboard Cite

Abstract-In this work we consider the problem of monitoring information streams for anomalies in a scalable and efficient manner. We study the problem in the context of network streams where the problem has received significant attention.Monitoring the empirical Shannon entropy of a feature in a network packet stream has previously been shown to be useful in detecting anomalies in the network traffic. Entropy is an information-theoretic statistic that measures the variability of the feature under consideration. Anomalous activity in network traffic can be captured by detecting changes in this variability.There are several challenges, however, in monitoring this statistic. Computing the statistic efficiently is non-trivial. Further, when monitoring multiple features, the streaming algorithms proposed previously would likely fail to keep up with the everincreasing channel bandwidth of network traffic streams. There is also the concern that an adversary could attempt to mask the effect of his attacks on variability by a mimicry attack disguising his traffic to mimic the distribution of normal traffic in the network, thus avoiding detection by an entropy monitoring sensor. Also, the high rate of false positives is a big problem with Intrusion Detection Systems, and the case of entropy monitoring is no different.In this work we propose a way to address the above challenges. First, we leverage recent progress in sketching algorithms to develop a distributed approach for computing entropic statistics accurately, at reasonable memory costs. Secondly, we propose monitoring not only regular entropy, but the related statistic of conditional entropy, as a more reliable measure in detecting anomalies. We implement our approach and evaluate it with real data collected at the link layer of an 802.11 wireless network.

show abstract

Functional Monitoring without Monotonicity

Cited by 67 publications

References 12 publications

When Distributed Computation Is Communication Expensive

When Distributed Computation Is Communication Expensive

Deviation Estimation between Distributed Data Streams

Distributed monitoring of conditional entropy for anomaly detection in streams

Contact Info

Product

Resources

About