A geometric approach to monitoring threshold functions over distributed data streams

Abstract

Monitoring data streams in a distributed system is the focus of much research in recent years. Most of the proposed schemes, however, deal with monitoring simple aggregated values, such as the frequency of appearance of items in the streams. More involved challenges, such as the important task of feature selection (e.g., by monitoring the information gain of various features), still require very high communication overhead using naive, centralized algorithms.We present a novel geometric approach which reduces monitoring the value of a function (visa-vis a threshold) to a set of constraints applied locally on each of the streams. The constraints are used to locally filter out data increments that do not affect the monitoring outcome, thus avoiding unnecessary communication. As a result, our approach enables monitoring of arbitrary threshold functions over distributed data streams in an efficient manner.We present experimental results on real-world data which demonstrate that our algorithms are highly scalable, and considerably reduce communication load in comparison to centralized algorithms. ACM Reference Format:Sharfman, I., Schuster, A., and Keren, D. 2007. A geometric approach to monitoring threshold functions over distributed data streams.

Decentralized detection of global threshold crossings using aggregation trees

“…For example, [1] suggests that an alert be raised based on for how long and/or by how much the variable stays above the threshold. A further alternative is raising an alert when the variable is within a specified interval around the threshold [1][2] [3]. Supporting the above notions of TCAs can be achieved through straightforward extensions of TCA-GAP.…”

“…Next, it checks whether a node needs to switch state from active to passive or vice versa. If the local threshold is 0 or if the partial aggregate is larger than the upper hysteresis threshold (or smaller than the lower threshold in negative mode), then the node, if not already active, switches to active state and resets the threshold of its children to 0 (lines [2][3][4][5]. If the node is active and the aggregate is below the lower hysteresis threshold (or above the upper hysteresis threshold in negative mode), then it switches to passive state and sets the threshold of its children (see section 3.3).…”

“…In [3], Sharfman et al present an approach where threshold crossings are detected for a real-valued function that is computed over the sum of vector-valued local variables. The authors present a weakly distributed and a strongly distributed versions of their approach to detection of threshold crossing.…”

Efficient parameter variation sampling for architecture simulations

“…For future work, we would like to further investigate the problem of dynamic fine-tuning of the grid-map and sample generation, in reaction to some (observed) changes in the parameters variation and component activity factors that may affect the validity of the experiments. Towards that, we will try to apply some of the techniques for streaming data management [25] in our context.…”

Communication-Efficient Distributed Online Prediction by Dynamic Model Synchronization

“…However, we cannot simply compute the divergence by centralizing all local models, because this would incur just as much communication as static full synchronization. Our strategy to overcome this problem is to first decompose the global condition δ(w) ≤ ∆ into a set of local conditions that can be monitored at their respective nodes without communication (see, e.g., Sharfman et al [2007]). Secondly, we define a resolution protocol Algorithm 2 Dynamic Synchronization Protocol Initialization:…”