GraphMP: I/O-Efficient Big Graph Analytics on a Single Commodity Machine

Abstract

Recent studies showed that single-machine graph processing systems can be as highly competitive as cluster-based approaches on large-scale problems. While several out-of-core graph processing systems and computation models have been proposed, the high disk I/O overhead could significantly reduce performance in many practical cases. In this paper, we propose GraphMP to tackle big graph analytics on a single machine. GraphMP achieves low disk I/O overhead with three techniques. First, we design a vertex-centric sliding window (VSW) computation model to avoid reading and writing vertices on disk. Second, we propose a selective scheduling method to skip loading and processing unnecessary edge shards on disk. Third, we use a compressed edge cache mechanism to fully utilize the available memory of a machine to reduce the amount of disk accesses for edges. Extensive evaluations have shown that GraphMP could outperform existing single-machine out-of-core systems such as GraphChi, X-Stream and GridGraph by up to 30, and can be as highly competitive as distributed graph engines like Pregel+, PowerGraph and Chaos.

I/O-Efficient Multi-Criteria Shortest Paths Query Processing on Large Graphs

“…Such a massive social network will consume about 7.2 TB to store the graph structural information, let alone the weights of multiple criteria. Although deploying more servers with larger memory and resorting to a cluster-based system is a solution, it is resource-inefficient as it requires a costly investment in a powerful computing infrastructure to handle big graphs [71]. For example, it requires a 64node cluster with 1024 processors and 16TB memory to process single-source shortest path (SSSP) queries on graphs with tens of billions of edges [81].…”