Scalable Splitting of Massive Data Streams

Abstract

Abstract. Scalable execution of continuous queries over massive data streams often requires splitting input streams into parallel sub-streams over which query operators are executed in parallel. Automatic stream splitting is in general very difficult, as the optimal parallelization may depend on application semantics. To enable application specific stream splitting, we introduce splitstream functions where the user specifies non-procedural stream partitioning and replication. For high-volume streams, the stream splitting itself becomes a performance bottleneck. A cost model is introduced that estimates the performance of splitstream functions with respect to throughput and CPU usage. We implement parallel splitstream functions, and relate experimental results to cost model estimates. Based on the results, a splitstream function called autosplit is proposed, which scales well for high degrees of parallelism, and is robust for varying proportions of stream partitioning and replication. We show how user defined parallelization using autosplit provides substantially improved scalability (L = 64) over previously published results for the Linear Road Benchmark.

Self-adaptive processing graph with operator fission for elastic stream processing

“…The conditions that must be met to guarantee the same results are described in Pollner et al (2015) . A similar model of fission operators is presented in Zeitler and Risch (2010) ; Wu and Liu (2014) , and Schneider et al (2012) . StreamCloud ( Gulisano et al, 2010 ) on the other hand, splits logical data into multiple physical data substreams that flow in parallel, preventing single-node bottlenecks.…”

Virtualizing Stream Processing

“…5.5) and that we can turn the originally centralized implementation of the Linear Road benchmark into a distributed implementation (see Sect. 5.6) to achieve one order of magnitude improvement in performance; a load factor of 64, which is equal to the fastest published results [30] of a highly optimized, distributed implementation. The other engine ported to ExoP is the Stanford STREAM system [20].…”

Knowledge-centric Analytics Queries Allocation in Edge Computing Environments

“…A multi-route optimizer is proposed in [5] exploting intra-and inter-stream correlations to produce effective partitions. The schemes in [11] and [26] propose the separation of streams into sets of sub-streams over which queries are executed in parallel.…”