Integrative Dynamic Reconfiguration in a Parallel Stream Processing Engine

Abstract

Load balancing, operator instance collocations and horizontal scaling are critical issues in Parallel Stream Processing Engines to achieve low data processing latency, optimized cluster utilization and minimized communication cost respectively. In previous work, these issues are typically tackled separately and independently. We argue that these problems are tightly coupled in the sense that they all need to determine the allocations of workloads and migrate computational states at runtime. Optimizing them independently would result in suboptimal solutions. Therefore, in this paper, we investigate how these three issues can be modeled as one integrated optimization problem. In particular, we first consider jobs where workload allocations have little effect on the communication cost, and model the problem of load balance as a Mixed-Integer Linear Program. Afterwards, we present an extended solution called ALBIC, which support general jobs. We implement the proposed techniques on top of Apache Storm, an open-source Parallel Stream Processing Engine. The extensive experimental results over both synthetic and real datasets show that our techniques clearly outperform existing approaches.

A Comprehensive Survey on Parallelization and Elasticity in Stream Processing

“…e authors use a tiered bin-packing problem, which allows for prioritizing operators, and solve the optimization problem using simulation-based latency estimation. Madsen et al [94] provide a solution to integrate load balancing, collocating (i.e., placement), and horizontal scaling (i.e., determining the parallelization degree of operators). ey model the problem as a Mixed-Integer Linear Program, solved with a heuristic greedy algorithm.…”

Optimal operator deployment and replication for elastic distributed data stream processing

“…The works most closely related to ours have been presented in Heinze et al() and Madsen et al Heinze et al() propose a model to estimate latency spikes caused by operator reallocations and use it to define a placement heuristic. It results in a solution that places only the newly added operators and minimizes the latency violations.…”

“…Moreover, it represents a benchmark against which heuristics can be compared. Madsen et al formulate a MILP optimization problem aimed to control load balancing and horizontal scaling, which works in combination with a heuristic in charge of collocating operators on computing nodes. Similarly to our work, their solution considers operator collocation (in our case, it follows from minimizing the response time) and state‐migration overheads.…”

“…Madsen et al 24 28 ; in the following, we focus on the issues that are more closely related to our work.…”

A survey of state management in big data processing systems

“…Thus far, state has been shown to be effective in several isolated application scenarios (i.e., fault tolerance, load balancing, elasticity). However, state can also be used to simultaneously address multiple scenarios simultaneously (e.g., scalability, fault tolerance [36,43,80,81,111]). It is in this scenario that multiobjective or integrative optimization (IO) arises.…”