Leaving stragglers at the window

Farhat, Omar; Bindra, Harsh; Daudjee, Khuzaima

doi:10.1145/3401025.3401732

Cited by 3 publications

(1 citation statement)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Watermarks take the form of punctuations [49,52] and are injected into the stream either (i) at the source, or (ii) by a specific operator that periodically emits them [4,12,20]. In both cases, applications decide on their implementation logic for generating watermarks.…”

Section: Watermarksmentioning

confidence: 99%

Klink: Progress-Aware Scheduling for Streaming Data Systems

Farhat

Daudjee

Querzoni

2021

Proceedings of the 2021 International Conference on Management of Data

Self Cite

View full text Add to dashboard Cite

Modern stream processing engines (SPEs) process large volumes of events propagated at high velocity through multiple queries.To improve performance, existing SPEs generally aim to minimize query output latency by minimizing, in turn, the propagation delay of events in query pipelines. However, for queries containing commonly used blocking operators such as windows, this scheduling approach can be inefficient. Watermarks are events popularly utilized by SPEs to correctly process window operators. Watermarks are injected into the stream to signify that no events preceding their timestamp should be further expected. Through the design and development of Klink, we leverage these watermarks to robustly infer stream progress based on window deadlines and network delay, and to schedule query pipeline execution that reflects stream progress. Klink aims to unblock window operators and to rapidly propagate events to output operators while performing judicious memory management. We integrate Klink into the popular open source SPE Apache Flink and demonstrate that Klink delivers significant performance gains over existing scheduling policies on benchmark workloads for both scale-up and scale-out deployments.

show abstract