A Performance Analysis of Fault Recovery in Stream Processing Frameworks

Abstract

Distributed stream processing frameworks have gained widespread adoption in the last decade because they abstract away the complexity of parallel processing. One of their key features is built-in fault tolerance. In this work, we dive deeper into the implementation, performance, and efficiency of this critical feature for four state-of-the-art frameworks. We include the established Spark Streaming and Flink frameworks and the more novel Spark Structured Streaming and Kafka Streams frameworks. We test the behavior under different types of faults and settings: master failure with and without high-availability setups, driver failures for Spark frameworks, worker failure with or without exactly-once semantics, application and task failures. We highlight differences in behavior during these failures on several aspects, e.g., whether there is an outage, downtime, recovery time, data loss, duplicate processing, accuracy, and the cost and behavior of different message delivery guarantees. Our results highlight the impact of framework design on the speed of fault recovery and explain how different use cases may benefit from different approaches. Due to their task-based scheduling approach, the Spark frameworks can recover quickly and in most cases without necessitating an application restart. Kafka Streams has the shortest downtime, while Flink can offer end-to-end exactly-once semantics at a low cost.

Benchmarking Scalability of Stream Processing Frameworks Deployed as Event-Driven Microservices in the Cloud

“…In the following, we give a brief overview of frameworks, particularly suited for implementing event-driven microservices and benchmarked in this study. For a detailed comparison of the framework's features, see the works of, for example, Hesse and Lorenz [42], Fragkoulis et al [18], and van Dongen [43].…”

“…OSPBench provides benchmarks for analyzing traffic sensor data. Besides evaluations of latency, throughput, and resource usage, van Dongen and van den Poel used OSPBench to also evaluate scalability [79] and fault recovery [80]. In contrast to most other benchmarks, OSPBench provides implementations for the rather new framework Kafka Streams, which is also evaluated in this study.…”

Scalability Benchmarking of Cloud-Native Applications Applied to Event-Driven Microservices

“…OSPBench provides benchmarks for analyzing traffic sensor data. Besides evaluations of latency, throughput, and resource usage, van Dongen and van den Poel used OSPBench to also evaluate scalability [vDvdP21b] and fault recovery [vDvdP21a]. In contrast to most other benchmarks, OSPBench provides implementations for the rather new framework Kafka Streams, which is also intensively studied in this thesis.…”