Signal Processing Challenges in Distributed Stream Processing Systems

Abstract

Distributed stream processing represents a novel computing paradigm where data, sensed externally and possibly preprocessed, is pushed asynchronously to various connected computing devices with heterogeneous capabilities for processing. It enables novel applications typically characterized by the need to process high-volume data streams in a timely and responsive fashion. Some example applications include sensor networks, location-tracking services, distributed speech recognition, and network management. Recent work in large-scale distributed stream processing tackle various research challenges in both the application domain as well as in the underlying system. The main focus of this paper is to highlight some of the signal processing challenges such a novel computing framework brings. We first briefly introduce the main concepts behind distributed stream processing. Then we define the notion of relevant information from two related information-theoretic approaches. Finally, we browse existing techniques for sensing and quantizing the information given the set of classification, detection and estimation tasks, which we refer to as task-driven signal processing. We also address some of the related unexplored research challenges.

Scheduling intense applications most 'surprising' first

“…Much work has examined aspects of streaming such as scalable architectures to exploit parallelism in data extraction [3]; query languages for expressing what information should be extracted and allowing the system to be reconfigured to respond to that query [12]; information theoretical aspects determining how a stream can be compressed while minimizing the loss of information [10]. Work on scheduling [4] in such systems has looked at applying cost functions to determine which tasks should be performed and in what order.…”