Algorithms for distributed functional monitoring

Abstract

We study what we call functional monitoring problems. We have k players each tracking their inputs, say player i tracking a multiset Ai(t) up until time t, and communicating with a central coordinator. The coordinator's task is to monitor a given function f computed over the union of the inputs ∪iAi(t), continuously at all times t. The goal is to minimize the number of bits communicated between the players and the coordinator. A simple example is when f is the sum, and the coordinator is required to alert when the sum of a distributed set of values exceeds a given threshold τ . Of interest is the approximate version where the coordinator outputs 1 if f ≥ τ and 0 if f ≤ (1 − )τ . This defines the (k, f, τ, ) distributed, functional monitoring problem.Functional monitoring problems are fundamental in distributed systems, in particular sensor networks, where we must minimize communication; they also connect to problems in communication complexity, communication theory, and signal processing. Yet few formal bounds are known for functional monitoring.We give upper and lower bounds for the (k, f, τ, ) problem for some of the basic f 's. In particular, we study frequency moments (F0, F1, F2). For F0 and F1, we obtain continuously monitoring algorithms with costs almost the same as their one-shot computation algorithms. However, for F2 the monitoring problem seems much harder. We give a carefully constructed multi-round algorithm that uses "sketch summaries" at multiple levels of detail and solves the (k, F2, τ, ) problem with communicationÕ(k 2 / + ( √ k/ ) 3 ). Since frequency moment estimation is central to other problems, our results have immediate applications to histograms, wavelet computations, and others. Our algorithmic techniques are likely to be useful for other functional monitoring problems as well.

Efficient Protocols for Distributed Classification and Optimization

“…Consider the case where each P i 's dataset arrives in a continuous stream; this is what is known as a distributed data stream (Cormode et al, 2008). Then applying results of (Cormode et al, 2010), we can continually maintain a sufficient random sample at the coordinator of size s ε communicating O((k + s ε,ν )d log |D|) words.…”

Uncovering Global Icebergs in Distributed Streams: Results and Implications

“…However, our goal is to come up with practical algorithms for both detecting and estimating the size of icebergs in real data sets. Recently, Cormode et al [16] proposed the problem of functional monitoring, where the local nodes continuously send updates only insofar as needed to satisfy some global constraint (e.g., detecting all the icebergs). Our work differs from theirs since we assume fixed measurement periods, which potentially allows us to have more communication-efficient mechanisms.…”

When Distributed Computation Is Communication Expensive

“…All protocols in the distributed streaming model are also valid protocols in our one-shot computational model, while our impossibility results in our one-shot computational model also apply to all protocols in the distributed streaming model. Example functions studied in the distributed streaming model include F 0 [7], F 2 (size of self join) [7,27], quantile and heavy-hitters [16], and the empirical entropy [3]. All of these problems have much lower communication cost if one allows an approximation of the output number x in a range [(1 − ε)x, (1 + ε)x], as mentioned above (the definition as to what ε is for the various problems differs).…”

“…In [7], a (1 + ε)-approximation algorithm (protocol) with O(k(log n + 1/ε 2 log 1/ε)) bits of communication was given in the distributed streaming model, which is also a protocol in the message-passing model. In a typical setting, we could have ε = 0.01, n = 10 9 and k = 1000, in which case the communication cost is about 6.6 × 10 7 bits 1 .…”