Benchmarking scalability of stream processing frameworks deployed as microservices in the cloud

ShuffleBench: A Benchmark for Large-Scale Data Shuffling Operations with Distributed Stream Processing Frameworks

“…Stream processing frameworks perform operations such as filterings, transformations, or aggregations in near-real time on continuous streams of data [19]. State-of-the-art frameworks are designed for high throughput and low-latency processing, while also scaling with massive amounts of data [9,16]. To address these requirements, they run in a distributed fashion on commodity hardware.…”

“…Table 1 provided an overview of these benchmarks and a comparison with ShuffleBench. For a systematic and comprehensive review of the literature on stream processing benchmarking, we refer to our recent studies [16,37].…”

“…Sustainable throughput [21] is defined as the maximum load a system can sustain without violating performance goals. Such a performance goal can be a limit on the event latency [21,34] or the maximum tolerable increase in the number of queued messages [13,16,34]. In ShuffleBench, sustainable throughput is measured by running multiple independent experiments, in which the generated load is increased from experiment to experiment and performance goals are evaluated [12].…”

“…For the ad-hoc throughput experiments, we generate 1 million records per second and monitor how many are processed per second by the frameworks. For the sustainable throughput experiments, we generate records with different frequencies and determine the maximum frequency at which the number of queued records in the Kafka input topic does not substantially increase over time (the performance goal, see our previous work for a detailed explanation of this method [16]). In the following, we first discuss the results of Flink, Kafka Streams, and Hazelcast as they allow for a straightforward interpretation, followed by a more detailed discussion of the results for Spark.…”

“…We illustrate this use case inspired by requirements of a large cloud observability platform, where potentially thousands or millions of stateful black-box software components have to receive and process selected data records. The literature is currently missing a well-defined evaluation method for this use case and previous work found that the performance of stream processing frameworks highly depends on the use case [16].…”