Fast failure recovery in distributed graph processing systems

Shen, Yanyan; Chen, Gang; Jagadish, H. V.; Lü, Wei; Ooi, Beng Chin; Tudor, Bogdan Marius

doi:10.14778/2735496.2735506

Cited by 36 publications

(24 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Single Failure: Instead of utilizing a new available node to substitute the failed node in [1,19], the healthy nodes are responsible for confined recovery in our approach once a single node failure is detected. Hence, the confined recovery process is applied in parallel on all the remaining healthy nodes.…”

Section: B Parallel Confined Recoverymentioning

confidence: 99%

“…However, this new node affords all the recomputation workload. A partitionbased reassignment algorithm [19] is proposed to accelerate the recovery process through simultaneous reduction of recovery communication costs and parallelization of the recovery computations. However, it relies on the statistics of the graph partition state kept by additional resources or components.…”

Section: B Failure Recoverymentioning

confidence: 99%

See 1 more Smart Citation

Efficient fault-tolerance for iterative graph processing on distributed dataflow systems

Holzemer

Kaul

et al. 2016

2016 IEEE 32nd International Conference on Data Engineering (ICDE)

View full text Add to dashboard Cite

Real-world graph processing applications often require combining the graph data with tabular data. Moreover, graph processing usually is part of a larger analytics workflow consiting of data preparation, analysis and model building, and model application. General-purpose distributed dataflow frameworks execute all steps of such workflows holistically. This holistic view enables these systems to reason about and automatically optimize the processing. Most big graph processing algorithms are iterative and incur a long runtime, as they require multiple passes over the data until convergence. Thus, fault tolerance and quick recovery from any intermittent failure at any step of the workflow are crucial for effective and efficient analysis. In this work, we propose a novel fault-tolerance mechanism for iterative graph processing on distributed data-flow systems with the objective to reduce the checkpointing cost and failure recovery time. Rather than writing checkpoints that block downstream operators, our mechanism writes checkpoints in an unblocking manner, without breaking pipelined tasks. In contrast to the typical unblocking checkpointing approaches (i.e., managing checkpoints independently for immutable datasets), we inject the checkpoints of mutable datasets into the iterative dataflow itself. Hence, our mechanism is iteration-aware by design. This simplifies the system architecture and facilitates coordinating the checkpoint creation during iterative graph processing. We achieve speedier recovery, i.e., confined recovery, by using the local log files on each node to avoid a complete re-computation from scratch. Our theoretical studies as well as our experimental analysis on Flink give further insight into our fault-tolerance strategies and show that they are more efficient than blocking checkpointing and complete recovery for iterative graph processing on dataflow systems.

show abstract

Section: B Parallel Confined Recoverymentioning

confidence: 99%

Section: B Failure Recoverymentioning

confidence: 99%

Efficient fault-tolerance for iterative graph processing on distributed dataflow systems

Holzemer

Kaul

et al. 2016

2016 IEEE 32nd International Conference on Data Engineering (ICDE)

View full text Add to dashboard Cite

show abstract

“…The simplest approach is to restart the entire computation. The most well-known alternatives include data materialization at synchronization boundaries [14,18,47], checkpointing the state of the entire computation either synchronously [33,36] or asynchronously [31], and restarting using lineage tracking and periodic checkpoints [40,49].…”

Section: Failure Handlingmentioning

confidence: 99%

Asynchronous and fault-tolerant recursive datalog evaluation in shared-nothing engines

2015

View full text Add to dashboard Cite

We present a new approach for data analytics with iterations. Users express their analysis in Datalog with bag-monotonic aggregate operators, which enables the expression of computations from a broad variety of application domains. Queries are translated into query plans that can execute in shared-nothing engines, are incremental, and support a variety of iterative models (synchronous, asynchronous, different processing priorities) and failure-handling techniques. The plans require only small extensions to an existing shared-nothing engine, making the approach easily implementable. We implement the approach in the Myria big-data management system and use our implementation to empirically study the performance characteristics of different combinations of iterative models, failure handling methods, and applications. Our evaluation uses workloads from a variety of application domains. We find that no single method outperforms others but rather that application properties must drive the selection of the iterative query execution model.

show abstract

“…Considering that there could be as many as 150,000 candidate n-grams per input (sentence) [5], the communication cost becomes prohibitively expensive. Another challenge of implementing distributed n-gram models is related to the network failure, which happens quite often in a distributed system with a large amount of network communication [23]. If some n-gram messages get lost, the model would produce inaccurate estimation of the probability.…”

Section: Introductionmentioning

confidence: 99%

A distributed system for large-scale n-gram language models at Tencent

Long¹,

Wang

Deng³

et al. 2019

Proc. VLDB Endow.

View full text Add to dashboard Cite

n-gram language models are widely used in language processing applications, e.g., automatic speech recognition, for ranking the candidate word sequences generated from the generator model, e.g., the acoustic model. Large n-gram models typically give good ranking results; however, they require a huge amount of memory storage. While distributing the model across multiple nodes resolves the memory issue, it nonetheless incurs a great network communication overhead and introduces a different bottleneck. In this paper, we present our distributed system developed at Tencent with novel optimization techniques for reducing the network overhead, including distributed indexing, batching and caching. They reduce the network requests and accelerate the operation on each single node. We also propose a cascade fault-tolerance mechanism which adaptively switches to small n-gram models depending on the severity of the failure. Experimental study on 9 automatic speech recognition (ASR) datasets confirms that our distributed system scales to large models efficiently, effectively and robustly. We have successfully deployed it for Tencent's WeChat ASR with the peak network traffic at the scale of 100 millions of messages per minute.

show abstract

Fast failure recovery in distributed graph processing systems

Cited by 36 publications

References 25 publications

Efficient fault-tolerance for iterative graph processing on distributed dataflow systems

Efficient fault-tolerance for iterative graph processing on distributed dataflow systems

Asynchronous and fault-tolerant recursive datalog evaluation in shared-nothing engines

A distributed system for large-scale n-gram language models at Tencent

Contact Info

Product

Resources

About