A lightweight infrastructure for graph analytics

Abstract

Several domain-specific languages (DSLs) for parallel graph analytics have been proposed recently. In this paper, we argue that existing DSLs can be implemented on top of a general-purpose infrastructure that (i) supports very fine-grain tasks, (ii) implements autonomous, speculative execution of these tasks, and (iii) allows application-specific control of task scheduling policies. To support this claim, we describe such an implementation called the Galois system.We demonstrate the capabilities of this infrastructure in three ways. First, we implement more sophisticated algorithms for some of the graph analytics problems tackled by previous DSLs and show that end-to-end performance can be improved by orders of magnitude even on power-law graphs, thanks to the better algorithms facilitated by a more general programming model. Second, we show that, even when an algorithm can be expressed in existing DSLs, the implementation of that algorithm in the more general system can be orders of magnitude faster when the input graphs are road networks and similar graphs with high diameter, thanks to more sophisticated scheduling. Third, we implement the APIs of three existing graph DSLs on top of the common infrastructure in a few hundred lines of code and show that even for power-law graphs, the performance of the resulting implementations often exceeds that of the original DSL systems, thanks to the lightweight infrastructure.

Locality Exists in Graph Processing: Workload Characterization on an Ivy Bridge Server

“…Galois [36] uses its own custom parallel runtime specifically designed to handle irregular fine-grained task parallelism. Algorithms implemented in Galois are free to use autonomous scheduling (no synchronization barriers), which should reduce the synchronization otherwise needed for high-diameter graphs.…”

Parallel k nearest neighbor graph construction using tree-based data structures

“…They go on to prove that a generalized form of data-parallelism exists in all algorithms and, depending on the algorithm's structure, this parallelism may be exploited by the program. They demon- strate their work with the Galois programming model [13].…”

“…The Galois framework is a parallel graph analytics infrastructure recently presented by Nguyen et al [13]. It is a work-item-based parallelization framework and provides a rich programming model that can handle graph computations.…”

“…The authors of Galois emphasize that a parallel program comprises an operator, schedule and parallel data structure and can be informally stated as: Parallel Program = Operator + Schedule + Parallel Data Structure [13]. A Galois operator is a user-defined function that operates on a graph node.…”

Big Graph Analytics Platforms

“…In addition, several systems including Ligra [Shun and Blelloch, 2013], Galois [Nguyen et al, 2013], Green-Marl DSL , etc., provide low-level graph programming frameworks that can handle nearly arbitrary graph computations. These frameworks often focus on specific classes of graph problems, and make a range of different assumptions about the computing environment, making them incomparable in many cases.…”