Cost-Aware Partitioning for Efficient Large Graph Processing in Geo-Distributed Datacenters

Abstract

Graph processing is an emerging computation model for a wide range of applications and graph partitioning is important for optimizing the cost and performance of graph processing jobs. Recently, many graph applications store their data on geo-distributed datacenters (DCs) to provide services worldwide with low latency. This raises new challenges to existing graph partitioning methods, due to the multi-level heterogeneities in network bandwidth and communication prices in geo-distributed DCs. In this paper, we propose an efficient graph partitioning method named Geo-Cut, which takes both the cost and performance objectives into consideration for large graph processing in geo-distributed DCs.Geo-Cut adopts two optimization stages. First, we propose a cost-aware streaming heuristic and utilize the one-pass streaming graph partitioning method to quickly assign edges to different DCs while minimizing inter-DC data communication cost. Second, we propose two partition refinement heuristics which identify the performance bottlenecks of geo-distributed graph processing and refine the partitioning result obtained in the first stage to reduce the inter-DC data transfer time while satisfying the budget constraint. Geo-Cut can be also applied to partition dynamic graphs thanks to its lightweight runtime overhead. We evaluate the effectiveness and efficiency of Geo-Cut using real-world graphs with both real geo-distributed DCs and simulations. Evaluation results show that Geo-Cut can reduce the inter-DC data transfer time by up to 79% (42% as the median) and reduce the monetary cost by up to 75% (26% as the median) compared to state-of-the-art graph partitioning methods with a low overhead.

PGPregel

“…There have been many graph processing systems [3,11,32,69,70,75] proposed for scalable and efficient graph executions in single DCs. Recently, there have been some studies focusing on the cost and performance optimizations for graph processing in geo-distributed DCs via graph partitioning [42,76,77]. However, such studies require vertex migration to perform graph partitioning, which introduces large inter-DC data movement and may violate privacy regulations when used in geo-distributed environments.…”

DMGA: A Distributed Shortest Path Algorithm for Multistage Graph

“…It can solve the multiple constraint and multiple objective graph partition problem on tera-scale graphs. Zhou et al [35] proposed Geo-Cut which uses a cost-aware streaming heuristic and two partition refinement heuristics to reduce the cost and data transfer time of geo-distributed data centres.…”

DMRO:A Deep Meta Reinforcement Learning-based Task Offloading Framework for Edge-Cloud Computing

“…• Application Partition: Since different tasks usually have different amounts of computation and communication, before performing task offloading operation, it is better to divide the task into a workflow with multiple associated subtasks or as a series of independent subtasks [105], and then offload the subtasks separately. Among them, some subtasks are executed on the IoT devices, the others are executed on the relatively powerful server, making full use of the server resources, thereby greatly reducing the load of the IoT devices and improving its endurance [86], [113].…”