Megaphone

Abstract

We design and implement Megaphone, a data migration mechanism for stateful distributed dataflow engines with latency objectives. When compared to existing migration mechanisms, Megaphone has the following differentiating characteristics: (i) migrations can be subdivided to a configurable granularity to avoid latency spikes, and (ii) migrations can be prepared ahead of time to avoid runtime coordination. Megaphone is implemented as a library on an unmodified timely dataflow implementation, and provides an operator interface compatible with its existing APIs. We evaluate Megaphone on established benchmarks with varying amounts of state and observe that compared to naïve approaches Megaphone reduces service latencies during reconfiguration by orders of magnitude without significantly increasing steady-state overhead.

To Migrate or Not to Migrate: An Analysis of Operator Migration in Distributed Stream Processing

“…Direct All-at-once state Checkpoint-assisted [51] No Checkpoint-assistance (Standard moving state) [46,47,50,85,93,141] [55, 61,62,99,105,107,110] Partial state [75] Indirect All-at-once state [142] Partial state Loose consistency (State shedding) [143] Strict consistency Checkpoint-assisted [58,78] Checkpoint-assisted [113] No Checkpoint-assistance [62,96] Stateless (Window-recreation) [62,99,103,141,144] Fig. 5: Parallel-track migration algorithms host while the operator is still processing on the old host.…”

“…Megaphone [75] is a state migration technique for migrating many keys in an efficient way to minimize latency spikes. In this case, the state is split into many equal-sized parts.…”

“…As such, static state transmission involves little or no overhead in query processing, and constitutes only one additional step in the algorithm. Note that a partial state movement algorithm might split the state into more than two parts, such as in Megaphone [75] and Meces [80].…”