Energy-Aware Scheduling of MapReduce Jobs for Big Data Applications

Mashayekhy, Lena; Nejad, Mahyar Movahed; Grosu, Daniel; Zhang, Quan; Shi, Weisong

doi:10.1109/tpds.2014.2358556

Cited by 117 publications

(41 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Similar to the partition scheme in GreenHDFS [16], MIA studied mixed workloads on MapReduce by assigning different SLAs to services [6], allowing interactive jobs to operate on a small pool of dedicated machines while less time-sensitive jobs run on the rest of the cluster in a batch fashion. Mashayekhy et al [26] proposed schedulers to assign server slots to minimizing energy consumed when executing MapReduce jobs. In all above algorithms, MapReduce is deployed on physical servers, rather than VMs.…”

Section: Related Workmentioning

confidence: 99%

Energy efficiency of VM consolidation in IaaS clouds

Teng

et al. 2016

J Supercomput

View full text Add to dashboard Cite

The energy efficiency of cloud computing has recently attracted a great deal of attention. As a result of raised expectations, cloud providers such as Amazon and Microsoft have started to deploy a new IaaS service, a MapReduce-style virtual cluster, to process data-intensive workloads. Considering that the IaaS provider supports multiple pricing options, we study batch-oriented consolidation and online 123 F. Teng et al. placement for reserved virtual machines (VMs) and on-demand VMs, respectively. For batch cases, we propose a DVFS-based heuristic TRP-FS to consolidate virtual clusters on physical servers to save energy while guarantee job SLAs. We prove the most efficient frequency that minimizes the energy consumption, and the upper bound of energy saving through DVFS techniques. More interestingly, this frequency only depends on the type of processor. FS can also be used in combination with other consolidation algorithms. For online cases, a time-balancing heuristic OTB is designed for on-demand placement, which can reduce the mode switching by means of balancing server duration and utilization. The experimental results both in simulation and using the Hadoop testbed show that our approach achieves greater energy savings than existing algorithms.

show abstract

Section: Related Workmentioning

confidence: 99%

Energy efficiency of VM consolidation in IaaS clouds

Teng

et al. 2016

J Supercomput

View full text Add to dashboard Cite

show abstract

“…Their proposed algorithm schedules jobs when electricity prices are sufficiently low and to places where the energy cost per unit work is low. Mashayekhy et al [18] proposed energy-aware scheduling algorithms for detailed task placement of MapReduce jobs. Their scheduling algorithms account for significant energy efficiency differences of different machines in a data center.…”

Section: Related Workmentioning

confidence: 99%

“…In the second phase, the algorithm finds the set of usersŨ, where each user's request is not greater than half of the available capacity of the PM p, for each resource by calling IS-FEASIBLE (lines [18][19][20][21][22]. Then, the algorithm calculates the bid densities of users inŨ (lines 23-24) according to a density metric defined as…”

Section: A Strategy-proof Approximation Mechanism For Pmrmmentioning

confidence: 99%

Physical Machine Resource Management in Clouds: A Mechanism Design Approach

Mashayekhy

Nejad

Grosu

2015

IEEE Trans. Cloud Comput.

Self Cite

View full text Add to dashboard Cite

We address the problem of physical machine resource management in clouds considering multiple types of physical machines and resources. We formulate this problem in an auction-based setting and design optimal and approximate strategy-proof mechanisms that solve it. Our proposed mechanisms consist of a winner determination algorithm that selects the users, provisions the virtual machines (VMs) to physical machines (PM), and allocates them to the selected users; and a payment function that determines the amount that each selected user needs to pay to the cloud provider. We prove that our proposed approximate winner determination algorithm satisfies the loser-independent property, making the approximate mechanism robust against strategic users who try to manipulate the system by changing other users' allocations. We show that our proposed mechanisms are strategy-proof, that is, the users do not have incentives to lie about their requested bundles of VM instances and their valuations. In addition, our proposed mechanisms are in alignment with green cloud computing strategies in which physical machines can be powered on or off to save energy. Our theoretical analysis shows that the proposed approximation mechanism has an approximation ratio of 3. We perform extensive experiments in order to investigate the performance of our proposed approximation mechanism compared to that of the optimal mechanism. Index Terms-cloud computing, mechanism design, energy efficient resource management. ✦ • L. Mashayekhy, M. M. Nejad, and D. Grosu are with the . His research interests include paralleland distributed systems, cloud computing, parallel algorithms, resource allocation, computer security, and topics at the border of computer science, game theory and economics. He has published more than ninety peer-reviewed papers in the above areas. He has served on the program and steering committees of several international meetings in parallel and distributed computing. He is a senior member of the ACM, the IEEE, and the IEEE Computer Society.

show abstract

“…Because of its prominence, increasingly number of studies have focused on improving the performance of Hadoop, and most of which are related to either task scheduling or job scheduling [18]- [23]. These studies can be generally classified into the following categories:…”

Section: Related Workmentioning

confidence: 99%

BOLAS: Bipartite-Graph Oriented Locality-Aware Scheduling for MapReduce Tasks

Xue

Gao

et al. 2015

2015 14th International Symposium on Parallel and Distributed Computing

View full text Add to dashboard Cite

Task scheduling is critical to reduce the makespan of MapReduce jobs. It is an effective approach for scheduling optimization by improving the data locality, which involves attempting to locate a task and its related data block on the same node. However, recent studies have been insufficient in addressing the locality issue. This paper proposes BOLAS, a MapReduce task scheduling algorithm, which models the scheduling process as a bipartite-graph matching problem trying best to assign data block to the nearest task. By considering the divergence of node performance of distribution of data blocks in MapReduce cluster, BOLAS can achieve a high degree of data locality, guarantee minimal data transfer during execution, and reduces a job's makespan subsequently. As a dynamic algorithm, BOLAS solves the model using Kuhn-Munkres optimal matching algorithm, and can be deployed in either homogeneous or heterogeneous environments. In this study, BOLAS was implemented as a plugin for Hadoop, and the experimental results indicate that BOLAS can localize nearly 100% of the map tasks and reduce the execution time by up to 67.1%.

show abstract

Energy-Aware Scheduling of MapReduce Jobs for Big Data Applications

Cited by 117 publications

References 33 publications

Energy efficiency of VM consolidation in IaaS clouds

Energy efficiency of VM consolidation in IaaS clouds

Physical Machine Resource Management in Clouds: A Mechanism Design Approach

BOLAS: Bipartite-Graph Oriented Locality-Aware Scheduling for MapReduce Tasks

Contact Info

Product

Resources

About