Influencing Factors in the Scalability of Distributed Stream Processing Jobs

Abstract

More and more use cases require fast, accurate, and reliable processing of large volumes of data. To do this, a distributed stream processing framework is needed which can distribute the load over several machines. In this work, we study and benchmark the scalability of stream processing jobs in four popular frameworks: Flink, Kafka Streams, Spark Streaming, and Structured Streaming. Besides that, we determine the factors that influence the performance and efficiency of scaling processing jobs with distinct characteristics. We evaluate horizontal, as well as vertical scalability. Our results show how the scaling efficiency is impacted by many factors including the initial cluster layout and direction of scaling, the pipeline design, the framework design, resource allocation, and data characteristics. Finally, we give some recommendations on how practitioners should undertake to scale their clusters.

An IoT-Based Services Infrastructure for Utility-Scale Distributed Solar Farms

“…While MQTT is ideal for real-time data communicate near the edge, it is not the most ideal choice for data streaming at the core where data from all distributed segments of the facility sinks. Data moves from MQTT to Kafka which is designed to handle high-velocity, high-volume real-time data [59,60], and is commonly used in real-time event detection systems [61]. Once data is available, the microservices-based design plays a large role in ensuring timely execution of various tasks [62,63].…”

Streaming vs. Functions: A Cost Perspective on Cloud Event Processing

“…Although including a cost model in cloud benchmarking studies is considered good scientific practice [37], in existing benchmarking studies on FaaS [24,[38][39][40][41] and DSP [4,42,43], cost evaluations can mainly be found for cloud functions, where the pay-perexecution pricing model has presented a significant paradigm shift.…”

Scalability Benchmarking of Cloud-Native Applications Applied to Event-Driven Microservices

“…In the following, we give a brief overview of modern distributed stream processing frameworks, particularly suited for implementing event-driven microservices. 6 For a detailed comparison, see the works of, for example, Hesse and Lorenz [HL15], Fragkoulis et al [FCK + 20], and van Dongen [vDon21].…”

“…OSPBench provides benchmarks for analyzing traffic sensor data. Besides evaluations of latency, throughput, and resource usage, van Dongen and van den Poel used OSPBench to also evaluate scalability [vDvdP21b] and fault recovery [vDvdP21a]. In contrast to most other benchmarks, OSPBench provides implementations for the rather new framework Kafka Streams, which is also intensively studied in this thesis.…”

“…We exclude Spark Structured Streaming since it does allow us to compute the lag trend via Kafka metrics [Von21a]. Within OSPBench, we select one of its aggregation pipelines [vDvdP21b] for our evaluation and create corresponding Theodolite benchmarks. These benchmarks use the number of generated messages per second as load type, the number of instances as resource type, and employ our lag trend SLO.…”