Fault-tolerant stream processing using a distributed, replicated file system

Abstract

We present SGuard, a new fault-tolerance technique for distributed stream processing engines (SPEs) running in clusters of commodity servers. SGuard is less disruptive to normal stream processing and leaves more resources available for normal stream processing than previous proposals. Like several previous schemes, SGuard is based on rollback recovery [18]: it checkpoints the state of stream processing nodes periodically and restarts failed nodes from their most recent checkpoints. In contrast to previous proposals, however, SGuard performs checkpoints asynchronously: i.e., operators continue processing streams during the checkpoint thus reducing the potential disruption due to the checkpointing activity. Additionally, SGuard saves the checkpointed state into a new type of distributed and replicated file system (DFS) such as GFS [22] or HDFS [9], leaving more memory resources available for normal stream processing. To manage resource contention due to simultaneous checkpoints by different SPE nodes, SGuard adds a scheduler to the DFS. This scheduler coordinates large batches of write requests in a manner that reduces individual checkpoint times while maintaining good overall resource utilization. We demonstrate the effectiveness of the approach through measurements of a prototype implementation in the Borealis [2] open-source SPE using HDFS [9] as the DFS.

A latency and fault-tolerance optimizer for online parallel query plans

“…We assume that individual operator partitions are deterministic, i.e., an operator partition produces an identical output when it processes the same input tuples in the same order. This is a common assumption [20,44,36,2,35,23] and most relational operators are deterministic. In a distributed system, however, the order in which input tuples reach an operator partition may not be deterministic.…”

“…As an optimization, operators can checkpoint only delta-changes of their state [11]. Other optimizations are also possible [11,19,23] and can be used with our framework.…”

“…In Section 4.2, however, we showed, how to efficiently implement three well-known fault-tolerance strategies for generic stateless and stateful operators. Existing libraries can also help with such implementation (e.g., [23]). Developers must also (b) model their operator costs within a pipelined query plan.…”

Detouring and replication for fast and reliable internet-scale stream processing

“…For this, we are implementing concurrent copy-onwrite data structures [14]. Further, due to other parallel, replicated data flows, any side-effect of operator migration is very likely to be hidden from the end-clients.…”

“…These techniques either execute all the operator replicas [4,10,20] or consistently copy the state of a subset of replicas onto other replicas [10,11,14]. In contrast to these solutions, our iFlow conducts detouring as soon as it notices a transmission problem.…”