Marabunta: Continuous Distributed Processing of Skewed Streams

Li, Bing; Zhang, Zhibin; Zheng, Tianqi; Zhong, Qiaoling; Huang, Qun; Cheng, Xueqi

doi:10.1109/ccgrid49817.2020.00-68

Cited by 3 publications

(1 citation statement)

References 16 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Direct All-at-once state Checkpoint-assisted [51] No Checkpoint-assistance (Standard moving state) [46,47,50,85,93,141] [55, 61,62,99,105,107,110] Partial state [75] Indirect All-at-once state [142] Partial state Loose consistency (State shedding) [143] Strict consistency Checkpoint-assisted [58,78] Checkpoint-assisted [113] No Checkpoint-assistance [62,96] Stateless (Window-recreation) [62,99,103,141,144] Fig. 5: Parallel-track migration algorithms host while the operator is still processing on the old host.…”

Section: Moving Statementioning

confidence: 99%

To Migrate or Not to Migrate: An Analysis of Operator Migration in Distributed Stream Processing

Volnes,

Plagemann,

Goebel

2024

IEEE Commun. Surv. Tutorials

View full text Add to dashboard Cite

One of the most important issues in distributed data stream processing systems is using operator migration to handle highly variable workloads cost-efficiently and adapt to the needs at any given time on demand. Operator migration is a complex process involving changes in the state and stream management of a running query, typically without any data loss, and with as little disruption to the execution as possible. This tutorial aims to introduce operator migration, explain the core elements of operator migration, and provide the reader with a good understanding of the design alternatives used in existing solutions. We developed a conceptual model to explain the fundamentals of operator migration and introduce a unified terminology, leading to a taxonomy of existing solutions. The conceptual model separates mechanisms, i.e., how to migrate, and policy, i.e., when to migrate. This separation is further applied to structure the description of existing solutions, offering the reader an algorithmic perspective on various design alternatives. To enhance our understanding of the impact of various design alternatives on migration mechanisms, we also conducted an empirical study that provides quantitative insights. The operator downtime for the naïve migration approach is almost 20 times longer than when applying an incremental checkpoint-based approach.

show abstract