Adaptive input admission and management for parallel stream processing

Balkesen, Çagri; Tatbul, Nesime; Özsu, M. Tamer

doi:10.1145/2488222.2488258

Cited by 41 publications

(42 citation statements)

References 24 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…algorithms [13,15,16,17,18] require the DSPE to support operator migration. Many DSPEs, such as Apache Storm, do not support migration, so we omit these algorithms from the evaluation.…”

Section: Discussionmentioning

confidence: 99%

“…The E-Store system explores the idea of handling hot tuples separately from cold tuples in the context of an elastic database management system [23]. Most existing load balancing techniques for DSPEs are analogous to key grouping with rebalancing [13,15,16,17,18]. Flux monitors the load of each operator, ranks servers by load, and migrates operators from the most loaded to the least loaded servers [13].…”

Section: Related Workmentioning

confidence: 99%

“…Similarly, (author?) [17] propose a frequency-aware hash-based partitioning to achieve load balance. (author?)…”

Section: Related Workmentioning

confidence: 99%

See 2 more Smart Citations

When two choices are not enough: Balancing at scale in Distributed Stream Processing

Nasir

Morales

Kourtellis

et al. 2016

2016 IEEE 32nd International Conference on Data Engineering (ICDE)

View full text Add to dashboard Cite

Abstract-Carefully balancing load in distributed stream processing systems has a fundamental impact on execution latency and throughput. Load balancing is challenging because real-world workloads are skewed: some tuples in the stream are associated to keys which are significantly more frequent than others. Skew is remarkably more problematic in large deployments: having more workers implies fewer keys per worker, so it becomes harder to "average out" the cost of hot keys with cold keys.We propose a novel load balancing technique that uses a heavy hitter algorithm to efficiently identify the hottest keys in the stream. These hot keys are assigned to d ≥ 2 choices to ensure a balanced load, where d is tuned automatically to minimize the memory and computation cost of operator replication. The technique works online and does not require the use of routing tables. Our extensive evaluation shows that our technique can balance real-world workloads on large deployments, and improve throughput and latency by 150% and 60% respectively over the previous state-of-the-art when deployed on Apache Storm.

show abstract

“…algorithms [13,15,16,17,18] require the DSPE to support operator migration. Many DSPEs, such as Apache Storm, do not support migration, so we omit these algorithms from the evaluation.…”

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

See 1 more Smart Citation

When two choices are not enough: Balancing at scale in Distributed Stream Processing

Nasir

Morales

Kourtellis

et al. 2016

2016 IEEE 32nd International Conference on Data Engineering (ICDE)

View full text Add to dashboard Cite

show abstract

“…However, it does not guarantee buffer limits on bursty workloads and does not account for the problem of large selections that have a delayed effect on the operator, as it is the case in our traffic monitoring operator. [37] tries to forecast the exact event arrival rate and assumes a fixed per-tuple processing cost when determining the optimal parallelization degree, which does not always hold, as we show with the operator profiles in our traffic monitoring scenario (cf. Section VI).…”

Section: ) Parallelization Degree Adaptationmentioning

confidence: 99%

Predictable Low-Latency Event Detection With Parallel Complex Event Processing

Mayer

Koldehofe

Rothermel

2015

IEEE Internet Things J.

View full text Add to dashboard Cite

The tremendous number of sensors and smart objects being deployed in the Internet of Things pose the potential for IT systems to detect and react to live-situations. For using this hidden potential, Complex Event Processing (CEP) systems offer means to efficiently detect event patterns (complex events) in the sensor streams and therefore help in realizing a "distributed intelligence" in the Internet of Things. With the increasing number of data sources and the increasing volume at which data is produced, parallelization of event detection is crucial to limit the time events need to be buffered before they actually can be processed. In this article, we propose a pattern-sensitive partitioning model for data streams that is capable of achieving a high degree of parallelism in detecting event patterns which formerly could only be consistently detected in a sequential manner or at a low parallelization degree. Moreover, we propose methods to dynamically adapt the parallelization degree to limit the buffering imposed on event detection in the presence of dynamic changes to the workload. Extensive evaluations of the system behavior show that the proposed partitioning model allows for a high degree of parallelism and that the proposed adaptation methods are able to meet a buffering limit for event detection under high and dynamic workloads.

show abstract

“…Nevertheless, the stream-based optimizations are applicable only at the level of multi-core nodes. Another approach consists in partitioning the input streams in an adaptive way considering information about the stream behavior, queries, load and external metadata [50]. Despite the fact that this solution targets the management of input streams, it has no support for the efficient transmission of data.…”

Section: Data Stream Management Systems (Dsms)mentioning

confidence: 99%