Composing Optimization Techniques for Vertex-Centric Graph Processing via Communication Channels

Abstract

Pregel's vertex-centric model allows us to implement many interesting graph algorithms, where optimization plays an important role in making it practically useful. Although many optimizations have been developed for dealing with different performance issues, it is hard to compose them together to optimize complex algorithms, where we have to deal with multiple performance issues at the same time. In this paper, we propose a new approach to composing optimizations, by making use of the channel interface, as a replacement of Pregel's message passing and aggregator mechanism, which can better structure the communication in Pregel algorithms. We demonstrate that it is convenient to optimize a Pregel program by simply using a proper channel from the channel library or composing them to deal with multiple performance issues. We intensively evaluate the approach through many nontrivial examples. By adopting the channel interface, our system achieves an all-around performance gain for various graph algorithms. In particular, the composition of different optimizations makes the S-V algorithm 2.20x faster than the current best implementation.

FastSV: A Distributed-Memory Connected Component Algorithm with Fast Convergence

“…MapReduce algorithms tend to perform poorly in the tightly-couple parallel systems our work targets, compared to the loosely-coupled architectures that are optimized for cloud workloads. The S-V PPA algorithm, due to the requirement of communicating between non-neighboring vertices, is only supported by several Pregel-like systems [1,2,30], and these frameworks tend to have limited scalability on multi-core clusters due to the lack of multi-threading support.…”