Consistency and Completeness

Wang, Guozhang; Chen, Lei; Dikshit, Ayusman; Gustafson, Jason; Chen, Boyang; Sax, Matthias J.; Roesler, J. F.; Blee-Goldman, Sophie; Cadonna, Bruno; Mehta, Apurva; Madan, Varun; Rao, Jun

doi:10.1145/3448016.3457556

Cited by 31 publications

(19 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Popular stream processing frameworks include Apache Flink [6], Hazelcast with its Jet engine [10], Apache Kafka Streams [31,38], and Apache Spark [39] with its Structured Streaming engine [4]. Although all these frameworks follow similar concepts, several differences in their design decisions, programming functionalities, and execution models can be noted.…”

Section: Distributed Stream Processingmentioning

confidence: 99%

“…Such messaging systems serve both as a scalable middleware between different systems and services as well as the necessary infrastructure to ensure fault tolerance. For this purpose, industry-grade messaging systems such as Apache Kafka [24,38] employ an immutable, sequentially appended log structure to store and replicate records across distributed nodes.…”

Section: Distributed Stream Processingmentioning

confidence: 99%

“…State-of-the-art distributed stream processing frameworks such as Spark [4,39], Flink [6], Kafka Streams [31,38], or Hazelcast with its Jet engine [10] have gained widespread adoption over the last years not only for building data analytics pipelines, but also for implementing core business logic in software-based organizations [9,22]. Such frameworks support software engineers in building highly scalable, reliable, and efficient applications that process continuous data streams of massive volume.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

ShuffleBench: A Benchmark for Large-Scale Data Shuffling Operations with Distributed Stream Processing Frameworks

Henning,

Vogel,

Leichtfried

et al. 2024

Proceedings of the 15th ACM/SPEC International Conference on Performance Engineering

View full text Add to dashboard Cite

Distributed stream processing frameworks help building scalable and reliable applications that perform transformations and aggregations on continuous data streams. This paper introduces Shuf-fleBench, a novel benchmark to evaluate the performance of modern stream processing frameworks. In contrast to other benchmarks, it focuses on use cases where stream processing frameworks are mainly employed for shuffling (i.e., re-distributing) data records to perform state-local aggregations, while the actual aggregation logic is considered as black-box software components. ShuffleBench is inspired by requirements for near real-time analytics of a large cloud observability platform and takes up benchmarking metrics and methods for latency, throughput, and scalability established in the performance engineering research community. Although inspired by a real-world observability use case, it is highly configurable to allow domain-independent evaluations. ShuffleBench comes as a ready-to-use open-source software utilizing existing Kubernetes tooling and providing implementations for four stateof-the-art frameworks. Therefore, we expect ShuffleBench to be a valuable contribution to both industrial practitioners building stream processing applications and researchers working on new stream processing approaches. We complement this paper with an experimental performance evaluation that employs ShuffleBench with various configurations on Flink, Hazelcast, Kafka Streams, and Spark in a cloud-native environment. Our results show that Flink achieves the highest throughput while Hazelcast processes data streams with the lowest latency.

show abstract

Section: Distributed Stream Processingmentioning

confidence: 99%

Section: Distributed Stream Processingmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

ShuffleBench: A Benchmark for Large-Scale Data Shuffling Operations with Distributed Stream Processing Frameworks

Henning,

Vogel,

Leichtfried

et al. 2024

Proceedings of the 15th ACM/SPEC International Conference on Performance Engineering

View full text Add to dashboard Cite

show abstract

“…Apache Kafka Streams. Kafka Streams [48,49] is a stream processing framework built on top of Apache Kafka. It is available as a Java library and, thus, aligns with the idea of incorporating stream processing in standalone microservices.…”

Section: Evaluated Stream Processing Frameworkmentioning

confidence: 99%

Benchmarking Scalability of Stream Processing Frameworks Deployed as Event-Driven Microservices in the Cloud

Henning,

Hasselbring

2023

Preprint

View full text Add to dashboard Cite

“…Handling late and out-of-order data on a data stream caught the attention of researchers a long time ago. Generally, there are three basic techniques to account for late data: punctuations [11], slack-time [12], a.k.a buffering, monotonic watermarks [8], [13], [14], [15], [16], order-agnostic processing [17], [18], [19], ordered processing [20], [21], [22], [23], [24], timestamp frontiers [25], [26]. Punctuations can reason completeness by assuming that no more events fulfilling a given condition or predicate will come in the future using the punctuation technique.…”

Section: Rq3mentioning

confidence: 99%

Keyed Watermarks: A Fine-grained Watermark Generation for Apache Flink

Yasser,

Arafa,

El-Helw

et al. 2023

Preprint

View full text Add to dashboard Cite

Big Data Stream processing engines, exemplified by tools like Apache Flink, employ windowing techniques to manage unbounded streams of events. Aggregating relevant data within Windows is important for event-time windowing due to its impact on result accuracy. A pivotal role in this process is attributed to watermarks, unique timestamps signifying event progression in time. Nonetheless, the existing watermark generation method within Apache Flink, operating at the input stream level, exhibits a bias towards faster sub-streams, causing the omission of events from slower counterparts. Our analysis determined that Apache Flink's standard watermark generation approach results in an approximate 33% data loss when 50% of median-proximate keys experience delays. Furthermore, this loss exceeds 37% in cases where 50% of randomly selected keys encounter delays. In this paper, we introduce a pioneering approach termed keyed watermarks to address data loss concerns and enhance data processing precision to a minimum of 99% in most scenarios. Our strategy facilitates distinct progress monitoring by creating individualized watermarks for each logical sub-stream (key). Within our investigation, we delineate the essential architectural and API modifications requisite for integrating keyed watermarks while also highlighting our experience in navigating the expansion of Apache Flink's extensive codebase. Moreover, we conduct a comparative evaluation between the efficacy of our approach and the conventional watermark generation technique concerning the accuracy of event-time tracking, the latency of watermark processing, and the growth of Flink's maintained state.

show abstract

Consistency and Completeness

Cited by 31 publications

References 29 publications

ShuffleBench: A Benchmark for Large-Scale Data Shuffling Operations with Distributed Stream Processing Frameworks

ShuffleBench: A Benchmark for Large-Scale Data Shuffling Operations with Distributed Stream Processing Frameworks

Benchmarking Scalability of Stream Processing Frameworks Deployed as Event-Driven Microservices in the Cloud

Keyed Watermarks: A Fine-grained Watermark Generation for Apache Flink

Contact Info

Product

Resources

About