Deviation Estimation between Distributed Data Streams

Abstract

Abstract-The analysis of massive data streams is fundamental in many monitoring applications. In particular, for networks operators, it is a recurrent and crucial issue to determine whether huge data streams, received at their monitored devices, are correlated or not as it may reveal the presence of malicious activities in the network system. We propose a metric, called codeviation, that allows to evaluate the correlation between distributed streams. This metric is inspired from classical metric in statistics and probability theory, and as such allows us to understand how observed quantities change together, and in which proportion. We then propose to estimate the codeviation in the data stream model. In this model, functions are estimated on a huge sequence of data items, in an online fashion, and with a very small amount of memory with respect to both the size of the input stream and the values domain from which data items are drawn. We give upper and lower bounds on the quality of the codeviation, and provide both local and distributed algorithms that additively approximates the codeviation among n data streams by using O ((1/ε) log(1/δ) (log N + log m)) bits of space for each of the n nodes, where N is the domain value from which data items are drawn, and m is the maximal stream's length. To the best of our knowledge, such a metric has never been proposed so far.

Byzantine-tolerant uniform node sampling service in large-scale networks

“…Most of the research done so far with this approach has mainly focused on computing functions or statistic measures with a given small error using a little amount of space with respect to the size of the input stream and the set from which items belong to. These include, as examples, the computation of the number of different data items in a given stream [19][20][21], the frequency moments [22], the most frequent data items [22,23], the entropy of the stream [24,25], or the proposition of a metric that allows to estimate a broad class of distance measures between any two massive streams [26], or the correlation among different streams [27].…”

Self-organized UAV-based Supervision and Connectivity: Challenges and Opportunities

“…Most existing research using this approach has focused on calculating statistical functions or measurements with a given ε error, using a poly-logarithmic memory amount in the size of the stream and the domain of the incoming data. These functions estimate, for example, the number of distinct data present in a given stream [9], [23], [29], the frequency moments [2], the most frequent elements [2], [19], [40], [37], the entropy of a flow [17], [34], the co-variance between several streams [3] or the relative entropy between a biased stream and a uniform stream [7], [4].…”