ADWISE: Adaptive Window-Based Streaming Edge Partitioning for High-Speed Graph Processing

Abstract

In recent years, the graph partitioning problem gained importance as a mandatory preprocessing step for distributed graph processing on very large graphs. Existing graph partitioning algorithms minimize partitioning latency by assigning individual graph edges to partitions in a streaming mannerat the cost of reduced partitioning quality. However, we argue that the mere minimization of partitioning latency is not the optimal design choice in terms of minimizing total graph analysis latency, i.e., the sum of partitioning and processing latency. Instead, for complex and long-running graph processing algorithms that run on very large graphs, it is beneficial to invest more time into graph partitioning to reach a higher partitioning quality -which drastically reduces graph processing latency. In this paper, we propose ADWISE, a novel window-based streaming partitioning algorithm that increases the partitioning quality by always choosing the best edge from a set of edges for assignment to a partition. In doing so, ADWISE controls the partitioning latency by adapting the window size dynamically at run-time. Our evaluations show that ADWISE can reach the sweet spot between graph partitioning latency and graph processing latency, reducing the total latency of partitioning plus processing by up to 23 − 47 percent compared to the state-of-the-art. Single-edgeAll-edge Fig. 1: Research gap -adaptive window-based streaming vertex-cut partitioning. NE[40]this paper due to its superior partitioning properties on realworld graphs compared to edge-cut partitioning [4]. In vertexcut partitioning, each vertex can reside on multiple partitions, i.e., can be replicated across the corresponding worker machines. However, a replicated vertex causes synchronization and communication overhead between the worker machines, inducing higher graph processing latency [2], [6], [7]. Hence, graph processing latency strongly correlates with partitioning quality, defined as the replication degree of vertices on the different worker machines. The problem of partitioning a graph optimally, i.e., with minimal vertex replication, is impracticable for large graphs due to its NP-hardness [8]. In literature, there are two basic approaches to practically address the partitioning problem: (i) single-edge streaming algorithms perform partitioning decisions on one edge at a time, minimizing the partitioning latency, or (ii) all-edge algorithms load the complete graph into memory and employ global placement heuristics to optimize the partitioning quality. The existing algorithms follow either of the methods: Figure 1 illustrates the landscape of stateof-the-art vertex-cut partitioning algorithms. Modern graph processing systems use streaming partitioning when loading massive graphs due to their superior scalability and minimal runtime complexity [4], [9].In this paper, we investigate whether it is always optimal to invest minimal partitioning latency as done by the established streaming partitioning algorithms. Clearly, there is a tradeoff between partitioning ...

Cost-Aware Partitioning for Efficient Large Graph Processing in Geo-Distributed Datacenters

“…Xu et al [23] propose a heterogeneity-aware streaming graph partitioning method, which considers the heterogeneous computing and communication abilities when placing graph vertices to different machines. Mayer et al [24] propose a window-based streaming graph partitioning method which selects the best edge from a set of edges during edge assignment to improve the quality of partitioning results. However, their method cannot be applied directly to the geo-distributed graph partitioning problem, due to the both inter-and intra-DC network heterogeneities in geo-distributed DCs.…”

Streaming Partitioning of RDF Graphs for Datalog Reasoning

“…Thus, the questions of how to partition RDF data effectively, and how this affects distributed reasoning, are still largely open. To answer the former, we draw inspiration from recent work on streaming graph partitioning [22,18,29,23,15,16] methods, which aim to produce good partitions while iterating over the graph edges a fixed number of times. The memory usage of these approaches is often determined by the number of vertices in the graph, which is usually at least an order of magnitude smaller than the number of edges; thus, the resource usage of such approaches is much smaller than for techniques such as METIS.…”

GAP: Genetic Algorithm Based Large-Scale Graph Partition in Heterogeneous Cluster

“…To achieve a tradeoff between partitioning latency and quality, slidewindow technology is applied in streaming algorithms. As a window-based streaming partitioning algorithm, ADWISE [20] encapsulates edges into a window whose size is automatically adapted at runtime according to the partition latency, and it chooses the best edge within the window for assignment to a partition by a scoring function considering load balance, degrees of vertices, and clustering coefficient. Different from ADWISE, WStream [21] utilizes a fixed-length window of vertices, and it employs a greedy strategy to assign the front vertex with its adjacent vertices to the most weighted partition.…”