Optimizing Geo-Distributed Data Analytics with Coordinated Task Scheduling and Routing

Zhao, Laiping; Yang, Yanan; Munir, Ali; Liu, Alex X.; Li, Yue; Qu, Wenyu

doi:10.1109/tpds.2019.2938164

Cited by 27 publications

(10 citation statements)

References 37 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Authors of [20] discuss a joint task scheduling and routing framework for geodistributed cloud frameworks. They propose a model to represent task-data dependencies and data-center dependencies as an augmented hyper-graph and propose many techniques to partition the hyper-graph in ways that minimize the data transfer volume.…”

Section: Literature Reviewmentioning

confidence: 99%

A Hierarchical Hadoop Framework to Process Geo-Distributed Big Data

Modica

Tomarchio

2022

BDCC

View full text Add to dashboard Cite

In the past twenty years, we have witnessed an unprecedented production of data worldwide that has generated a growing demand for computing resources and has stimulated the design of computing paradigms and software tools to efficiently and quickly obtain insights on such a Big Data. State-of-the-art parallel computing techniques such as the MapReduce guarantee high performance in scenarios where involved computing nodes are equally sized and clustered via broadband network links, and the data are co-located with the cluster of nodes. Unfortunately, the mentioned techniques have proven ineffective in geographically distributed scenarios, i.e., computing contexts where nodes and data are geographically distributed across multiple distant data centers. In the literature, researchers have proposed variants of the MapReduce paradigm that obtain awareness of the constraints imposed in those scenarios (such as the imbalance of nodes computing power and of interconnecting links) to enforce smart task scheduling strategies. We have designed a hierarchical computing framework in which a context-aware scheduler orchestrates computing tasks that leverage the potential of the vanilla Hadoop framework within each data center taking part in the computation. In this work, after presenting the features of the developed framework, we advocate the opportunity of fragmenting the data in a smart way so that the scheduler produces a fairer distribution of the workload among the computing tasks. To prove the concept, we implemented a software prototype of the framework and ran several experiments on a small-scale testbed. Test results are discussed in the last part of the paper.

show abstract

Section: Literature Reviewmentioning

confidence: 99%

A Hierarchical Hadoop Framework to Process Geo-Distributed Big Data

Modica

Tomarchio

2022

BDCC

View full text Add to dashboard Cite

show abstract

“…Tetrium [3] jointly considers the heterogeneity of computing in tandem with network resources in designing the placement strategy of the involved map and reduce tasks. To save the amount of data transmission and reduce the makespan, HPS+ [22] proposes a new resource allocation algorithm. Literature [26] provides a federation file system atop of geo-distributed edge servers, and proposes a generic job and resource-aware data storage and placement algorithm (JRAP).…”

Section: Related Workmentioning

confidence: 99%

“…• HPS+ [22]: a new resource allocation algorithm that orchestrates the replica selection and task placement to minimize the volume of transferred data. Evaluation metrics: The following two metrics are used to measure the performance:…”

Section: A Settings Of Evaluationmentioning

confidence: 99%

“…Actually, not all geo-distributed data analytics jobs are required to be executed as soon as possible and obtain real-time results. There are many jobs that just need to be completed before a given deadline [20], [21], such as geo-distributed batch jobs [19] and geo-distributed scientific computations [22]. With this in mind, we propose to minimize the JCC of geo-distributed analytic jobs which have certain deadline requirements.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Geo-Distributed IoT Data Analytics With Deadline Constraints Across Network Edge

Chen

Luo

Ren

et al. 2022

IEEE Internet Things J.

View full text Add to dashboard Cite

Owing to the advancement of the Internet of Things (IoT) and 5G mobile technologies, various IoT devices produce massive data which is usually transferred to nearby sites, such as edge nodes or datacenters. Many large-scale IoT applications need to analyze the data distributed across multiple sites to obtain final results. A dominant challenge of this type of data analytics is the heterogeneities of resource capacities across geodistributed sites. In this paper, we find that the resource capacity as well as the resource price differ among sites, and the price heterogeneity has a significant impact on geo-distributed IoT data analytics. Thus, each geo-distributed IoT data analytics job prefers to minimize the job execution cost while guaranteeing its deadline requirement under the resource constraints of involved sites. Specifically, we propose to jointly consider the resource heterogeneities of both capacity and price, and minimize the cost of each job before its deadline. We characterize this optimization problem as a Quadratically Constrained Quadratic Programming problem. To tackle such an NP-hard problem, we propose the MCGL method, which calculates a task placement solution by gradient adjustment strategy according to the remarkable negative correlation relationship between job completion time and job completion cost of geo-distributed IoT data analytics job. The task placement strategy can optimize resource cost with respect to the deadline requirement of any geo-distributed data analytics job. The trace-driven evaluations indicate that MCGL significantly reduces the total cost compared with existing methods; moreover, they satisfy the deadline constraints simultaneously.

show abstract

“…If links become congested, then the latency will increase and throughput will suffer. A key issue is how to allocate enough bandwidth to each application without causing congestion on the network [40].…”

Section: Introductionmentioning

confidence: 99%

Target Location Problem for Multi-commodity Flow

Liu¹,

Pan²,

Wang³

2020

Preprint

View full text Add to dashboard Cite

Motivated by scheduling in Geo-distributed data analysis, we propose a target location problem for multi-commodity flow (LoMuF for short). Given commodities to be sent from their resources, LoMuF aims at locating their targets so that the multi-commodity flow is optimized in some sense. LoMuF is a combination of two fundamental problems, namely, the facility location problem and the network flow problem. We study the hardness and algorithmic issues of the problem in various settings. The findings lie in three aspects. First, a series of NP-hardness and APX-hardness results are obtained, uncovering the inherent difficulty in solving this problem. Second, we propose an approximation algorithm for general undirected networks and an exact algorithm for undirected trees, which naturally induce efficient approximation algorithms on directed networks. Third, we observe separations between directed networks and undirected ones, indicating that imposing direction on edges makes the problem strictly harder. These results show the richness of the problem and pave the way to further studies.

show abstract

Optimizing Geo-Distributed Data Analytics with Coordinated Task Scheduling and Routing

Cited by 27 publications

References 37 publications

A Hierarchical Hadoop Framework to Process Geo-Distributed Big Data

A Hierarchical Hadoop Framework to Process Geo-Distributed Big Data

Geo-Distributed IoT Data Analytics With Deadline Constraints Across Network Edge

Target Location Problem for Multi-commodity Flow

Contact Info

Product

Resources

About