Optimizing checkpoint‐based fault‐tolerance in distributed stream processing systems: Theory to practice

Abstract

Fault-tolerance is an essential part of a stream processing system that guarantees data analysis could continue even after failures. State-of-the-art distributed stream processing systems use checkpointing to support fault-tolerance for stateful computations where the state of the computations is periodically persisted. However, the frequency of performing checkpoints impacts the performance (utilization, latency, and throughput) of the system as the checkpointing process consumes resources and time that can be used for actual computations. In practice, systems are often configured to perform checkpoints based on crude values ignoring factors such as checkpoint and restart costs, leading to suboptimal performance. In our previous work, we proposed a theoretical optimal checkpoint interval that maximizes the system utilization for stream processing systems to minimize the impact of checkpointing on system performance.In this article, we investigate the practical benefits of our proposed theoretical optimal by conducting experiments in a real-world cloud setting using different streaming applications; we use Apache Flink, a well-known stream processing system for our experiments. The experiment results demonstrate that an optimal interval can achieve better utilization, confirming the practicality of the theoretical model when applied to real-world applications. We observed utilization improvements from 10% to 200% for a range of failure rates from 0.3 failures per hour to 0.075 failures per minute. Moreover, we explore how performance measures: latency and throughput are affected by the optimal interval.Our observations demonstrate that significant improvements can be achieved using the optimal interval for both latency and throughput.

Scalable Communication-Induced Checkpointing Protocol with Little Overhead for Distributed Computing Environments

“…In a distributed system consisting of a set of n processes following the crash failure model, the processes exchange messages with each other through reliable FIFO communication channels [17]. Each process p takes its i-th local checkpoint (i ≥ 0), Ck i p , to record its current state in stable storage for recovery.…”