Pregel algorithms for graph connectivity problems with performance guarantees

Yan, Da; Cheng, James; Xing, Kai; Lu, Yi; Ng, Wilfred; Bu, Yingyi

doi:10.14778/2733085.2733089

Cited by 90 publications

(99 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The Awerbuch-Shiloach (AS) algorithm [4] is a simplification of SV by using a different termination condition. Transforming the complete SV or AS to distributedmemory is possible [6,27], but the detection of stagnant trees in SV's unconditional hooking step is in fact not suitable for a distributed-memory implementation, which introduces considerable computation and communication cost. Therefore, we based our distributedmemory FastSV on a simplified SV algorithm preserving only the essential steps and introduce efficient hooking steps to ensure fast convergence in practice.…”

Section: Executionmentioning

confidence: 99%

“…ParConnect [16] is another distributed-memory algorithm that adaptively uses parallel BFS and SV and dynamically selects which method to use. For other software architectures, there are Hash-Min [21] for MapReduce systems and S-V PPA [27] for vertex-centric message passing systems [17]. MapReduce algorithms tend to perform poorly in the tightly-couple parallel systems our work targets, compared to the loosely-coupled architectures that are optimized for cloud workloads.…”

Section: Executionmentioning

confidence: 99%

See 1 more Smart Citation

FastSV: A Distributed-Memory Connected Component Algorithm with Fast Convergence

Zhang¹,

Azad²,

Hu³

2020

Proceedings of the 2020 SIAM Conference on Parallel Processing for Scientific Computing

View full text Add to dashboard Cite

This paper presents a new distributed-memory algorithm called FastSV for finding connected components in an undirected graph. Our algorithm simplifies the classic Shiloach-Vishkin algorithm and employs several novel and efficient hooking strategies for faster convergence. We map different steps of FastSV to linear algebraic operations and implement them with the help of scalable graph libraries. FastSV uses sparse operations to avoid redundant work and optimized MPI communication to avoid bottlenecks. The resultant algorithm shows high-performance and scalability as it can find the connected components of a hyperlink graph with over 134B edges in 30 seconds using 262K cores on a Cray XC40 supercomputer. FastSV outperforms the state-ofthe-art algorithm by an average speedup of 2.21× (max 4.27×) on a variety of real-world graphs.

show abstract

Section: Executionmentioning

confidence: 99%

Section: Executionmentioning

confidence: 99%

FastSV: A Distributed-Memory Connected Component Algorithm with Fast Convergence

Zhang¹,

Azad²,

Hu³

2020

Proceedings of the 2020 SIAM Conference on Parallel Processing for Scientific Computing

View full text Add to dashboard Cite

show abstract

“…Here, efficient algorithms refer to the ones satisfying the constraints for Practical Pregel Algorithms in [18]. A Pregel algorithm might consist of one or many supersteps.…”

Section: Parallel Evaluation Over Pregelmentioning

confidence: 99%

Let High-level Graph Queries Be Parallel Efficient: An Approach Over Structural Recursion On Pregel

Tung

Meng

et al. 2016

Journal of Information Processing

View full text Add to dashboard Cite

Graphs play an important role today in managing big data. Supporting declarative graph queries is one of the most crucial parts for efficiently manipulating graph databases. Structural recursion has been studied for graph querying and graph transformations. However, most of the previous studies about graph structural recursion do not exploit in practical the power of parallel computing. The bulk semantics, which is used for parallel evaluation of structural recursion, still impose many constraints that limit the performance of querying in parallel. In this paper, we propose a framework that systematically generates structural recursive functions from high-level declarative graph queries, then the generated functions are evaluated efficiently on our framework on top of the Pregel model. Therefore, the complexity in developing efficient structural recursive functions is relaxed by our solution.

show abstract

“…It is based on the BSP model [25] and adopts the vertex-centric paradigm with explicit messages to support scalable big graph processing. Pregel's vertex-centric model has demonstrated its usefulness in implementing many interesting graph algorithms [16], [18], [21], [30], and imposed influence over the design of Pregel-like systems, such as Giraph [1], GPS [19], Mizan [11], Pregel+ [2].…”

Section: Introductionmentioning

confidence: 99%

Composing Optimization Techniques for Vertex-Centric Graph Processing via Communication Channels

Zhang

2019

2019 IEEE International Parallel and Distributed Processing Symposium (IPDPS)

View full text Add to dashboard Cite

Pregel's vertex-centric model allows us to implement many interesting graph algorithms, where optimization plays an important role in making it practically useful. Although many optimizations have been developed for dealing with different performance issues, it is hard to compose them together to optimize complex algorithms, where we have to deal with multiple performance issues at the same time. In this paper, we propose a new approach to composing optimizations, by making use of the channel interface, as a replacement of Pregel's message passing and aggregator mechanism, which can better structure the communication in Pregel algorithms. We demonstrate that it is convenient to optimize a Pregel program by simply using a proper channel from the channel library or composing them to deal with multiple performance issues. We intensively evaluate the approach through many nontrivial examples. By adopting the channel interface, our system achieves an all-around performance gain for various graph algorithms. In particular, the composition of different optimizations makes the S-V algorithm 2.20x faster than the current best implementation.

show abstract

Pregel algorithms for graph connectivity problems with performance guarantees

Cited by 90 publications

References 22 publications

FastSV: A Distributed-Memory Connected Component Algorithm with Fast Convergence

FastSV: A Distributed-Memory Connected Component Algorithm with Fast Convergence

Let High-level Graph Queries Be Parallel Efficient: An Approach Over Structural Recursion On Pregel

Composing Optimization Techniques for Vertex-Centric Graph Processing via Communication Channels

Contact Info

Product

Resources

About