GraphD: Distributed Vertex-Centric Graph Processing Beyond the Memory Limit

Yan, Da; Huang, Yuzhen; Liu, Miao; Chen, Hongzhi; Cheng, James; Wu, Huanhuan; Zhang, Chengcui

doi:10.1109/tpds.2017.2743708

Cited by 37 publications

(21 citation statements)

References 28 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…If a graph processing system uses an external storage, there is another I/O cost to read the partitioned graphs from the external storage. For example, systems like HybridGraph, GraphD [41] and Pregelix store the partitioned subgraphs in local disks of workers, and load the subgraphs in every iteration to avoid scalability issues such as out-ofmemory error. Reading the subgraphs from external storage causes massive disk I/Os.…”

Section: Costs Of Distributed Graph Processing Systemsmentioning

confidence: 99%

FlexGraph: Flexible partitioning and storage for scalable graph mining

Park

Kang

2020

PLoS ONE

View full text Add to dashboard Cite

How can we analyze large graphs such as the Web, and social networks with hundreds of billions of vertices and edges? Although many graph mining systems have been proposed to perform various graph mining algorithms on such large graphs, they have difficulties in processing Web-scale graphs due to massive communication and I/O costs caused by communication between workers, and reading subgraphs repeatedly. In this paper, we propose FlexGraph, a scalable distributed graph mining method reducing the costs by exploiting properties of real-world graphs. FlexGraph significantly decreases the communication cost, which is the main bottleneck of distributed systems, by exploiting different edge placement policies based on types of vertices. Furthermore, we propose a flexible storage format to reduce I/O costs when reading input graph repeatedly. Experiments show that FlexGraph succeeds in processing up to 64× larger graphs than existing distributed memory-based graph mining methods, and consistently outperforms previous disk-based graph mining methods.

show abstract

Section: Costs Of Distributed Graph Processing Systemsmentioning

confidence: 99%

FlexGraph: Flexible partitioning and storage for scalable graph mining

Park

Kang

2020

PLoS ONE

View full text Add to dashboard Cite

show abstract

“…In-Memory Out-of-Core (use HDD if not indicated) In-Memory Out-of-Core In-Memory Out-of-Core Approaches Ligra [7] Galois [8] GraphMat [9] Polymer [10] GraphChi [31] X-Stream [32] VENUS [33] GridGraph [34] GraphMP (Use SSD) FlashGraph [35] TurboGraph [36] Medusa [11] Gunrock [13] MapGraph [14] gGraph [15] (Use SSD) GTS [16] GGraph [15] Pregel-like: [20] [21] [22] [23] [24] GAS: [25] [26] [27] SpMV: [28] [29] (Use HDD) GraphD [37] Chaos [38] Pregelix [ GraphChi, the edge-centric scatter-gather (ESG) model of X-Stream, the vertex-centric streamlined processing (VSP) model of VENUS, and the dual sliding windows (DSW) model of GridGraph. These approaches try to exploit the sequential bandwidth of hard disks and to reduce the amount of disk accesses.…”

Section: Single Machine (Cpu) Single Machine (Gpu) Cluster Data Storagementioning

confidence: 99%

GraphMP: I/O-Efficient Big Graph Analytics on a Single Commodity Machine

Sun

Wen

Duong

et al. 2020

IEEE Trans. Big Data

View full text Add to dashboard Cite

Recent studies showed that single-machine graph processing systems can be as highly competitive as cluster-based approaches on large-scale problems. While several out-of-core graph processing systems and computation models have been proposed, the high disk I/O overhead could significantly reduce performance in many practical cases. In this paper, we propose GraphMP to tackle big graph analytics on a single machine. GraphMP achieves low disk I/O overhead with three techniques. First, we design a vertex-centric sliding window (VSW) computation model to avoid reading and writing vertices on disk. Second, we propose a selective scheduling method to skip loading and processing unnecessary edge shards on disk. Third, we use a compressed edge cache mechanism to fully utilize the available memory of a machine to reduce the amount of disk accesses for edges. Extensive evaluations have shown that GraphMP could outperform existing single-machine out-of-core systems such as GraphChi, X-Stream and GridGraph by up to 30, and can be as highly competitive as distributed graph engines like Pregel+, PowerGraph and Chaos.

show abstract

“…After Pregel, many systems were proposed to optimize it. Giraph [21] is the open-source counterpart to Pregel; GraphX DBpedia 265 620 885 YAGO 651 1,822 2,473 Soc-Academia 187 479 666 Soc-YouTube 233 779 1,012 RoadNet-TX 2,287 3,039 5,326 RoadNet-CA 2,644 3,489 6,133 TABLE 2: Offline Performance (in min) [9] is Spark's graphs computing API, which also implements the Pregel operator; Pregel+ [25] proposes integration mirroring and message combining as well as a request-response mechanism; Quegel [28] extends the vertex-centric model to the query-centric model; TurboGraph++ [11] discusses how to process large graphs by exploiting external memory without compromising efficiency; GraphD [26] adopts a semistreaming model to avoid scanning the whole graph in each superstep, where only a portion of the vertex states are maintained in the main memory and edges and messages are streamed on the local disk; G-thinker [24] extends the vertexcentric model to the subgraph-centric model for computeintensive graph mining workloads.…”

Section: B Distributed Graph Processing Systemsmentioning

confidence: 99%

“…When graph models are used in an increasing number of applications, as one of the most classic problems in a graph, single-source shortest path length (SSSP length) queries have been studied for more than half a century and have received increasing attention [6], [16], [26]. Given a graph and a source vertex, an SSSP length query finds the distance from the source vertex to each vertex in the graph.…”

Section: Introductionmentioning

confidence: 99%

Optimizing Distance Computation in Distributed Graph Systems

Wang

Peng

et al. 2020

IEEE Access

View full text Add to dashboard Cite

Given a large graph, such as a social network or a knowledge graph, a fundamental query is how to find the distance from a source vertex to another vertex in the graph. As real graphs become very large and many distributed graph systems, such as Pregel [16], Pregel+ [25], Giraph [21], and GraphX [9], are proposed, how to employ distributed graph systems to process single-source distance queries should attract more attention. In this paper, we propose a landmark-based framework to optimize the distance computation over distributed graph systems. We also use a measure called set betweenness to select the optimal set of landmarks for distance computation. Although we can prove that selecting the optimal set of landmarks is NP-hard, we propose a heuristic distributed algorithm that can guarantee the approximation ratio. Experiments on large real graphs confirm the superiority of our methods.

show abstract

GraphD: Distributed Vertex-Centric Graph Processing Beyond the Memory Limit

Cited by 37 publications

References 28 publications

FlexGraph: Flexible partitioning and storage for scalable graph mining

FlexGraph: Flexible partitioning and storage for scalable graph mining

GraphMP: I/O-Efficient Big Graph Analytics on a Single Commodity Machine

Optimizing Distance Computation in Distributed Graph Systems

Contact Info

Product

Resources

About