Resource-Aware Adaptive Scheduling for MapReduce Clusters

Polo, Jordà; Castillo, Claris; Carrera, David; Becerra, Yolanda; Whalley, Ian; Steinder, Małgorzata; Torres, Jordi; Ayguadé, Eduard

doi:10.1007/978-3-642-25821-3_10

Cited by 111 publications

(78 citation statements)

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this algorithm, two important concepts are used, which are Efficiency and Fairness [8], [14]. However, a Map-Reduce scheduler is responsible to assign each task to an appropriate machine along with the consideration of both Efficiency and Fairness.…”

Section: Algorithm Descriptionmentioning

confidence: 99%

PRISM: Fine-Grained Phase and Resource Information-aware Scheduler for Map-Reduce

Mahendrakar¹,

Patil²

2017

IJCA

View full text Add to dashboard Cite

In recent years, Map Reduce has become a popular model with regard to data-intensive computation. Map Reduce can significantly reduce the execution time of data-intensive jobs. In order to achieve this objective, Map Reduce breaks down each job into small map and reduce tasks and executes them in parallel across a large number of machines. However, existing solutions mainly focus on scheduling at the task-level, which offer sub-optimal job performance, because tasks may have resource requirements which may vary during their lifetime. This makes it difficult for existing system's task-level schedulers to effectively utilize available resources in order to reduce job execution time.To avoid this limitation, PRISM is introduced. PRISM stands for Phase and Resource Information-aware Scheduler for Map-Reduce. PRISM consists of various clusters that perform resource-aware scheduling at the level of phases. PRISM can be defined as a fine-grained resource-aware Map Reduce scheduler that divides tasks into phases. Here, each phase has a constant resource usage profile, so that not a single phase suffers from starvation. PRISM also offers high resource utilization and provides 1:3x improvements in job running time as compared to the current Hadoop schedulers.

show abstract

Section: Algorithm Descriptionmentioning

confidence: 99%

PRISM: Fine-Grained Phase and Resource Information-aware Scheduler for Map-Reduce

Mahendrakar¹,

Patil²

2017

IJCA

View full text Add to dashboard Cite

show abstract

“…Reference [25] introduces a Quincy scheduler to achieve data locality. Several recent proposals, such as resource-aware adaptive scheduling [26] and cost effective resource provisioning [27], have introduced resource-aware job schedulers to the MapReduce framework. Reference [28] mentions the problem of task assignment with the consideration of the data locality in cloud computing.…”

Section: Related Workmentioning

confidence: 99%

Optimized Speculative Execution to Improve Performance of MapReduce Jobs on Virtualized Computing Environment

Yang

Zhang

2017

Mathematical Problems in Engineering

View full text Add to dashboard Cite

Recently, virtualization has become more and more important in the cloud computing to support efficient flexible resource provisioning. However, the performance interference among virtual machines may affect the efficiency of the resource provisioning. In a virtualized environment, where multiple MapReduce applications are deployed, the performance interference can also affect the performance of the Map and Reduce tasks resulting in the performance degradation of the MapReduce jobs. Then, in order to ensure the performance of the MapReduce jobs, a framework for scheduling the MapReduce jobs with the consideration of the performance interference among the virtual machines is proposed. The core of the framework is to identify the straggler tasks in a job and back up these tasks to make the backed up one overtake the original tasks in order to reduce the overall response time of the job. Then, to identify the straggler task, this paper uses a method for predicting the performance interference degree. A method for scheduling the backing-up tasks is presented. To verify the effectiveness of our framework, a set of experiments are done. The experiments show that the proposed framework has better performance in the virtual cluster compared with the current speculative execution framework.

show abstract

“…The former is used to decide task placement on tasktrackers over time, while the latter is used to estimate the number of tasks to be run in parallel for each job in order to meet performance objectives, expressed in the form of completion time goals. Job performance management has been extensively evaluated and validated in our previous work, presented as the Adaptive Scheduler [6] [7]. In this paper we extend the resource availability awareness of the scheduler when the MapReduce jobs are collocated with other timevarying workloads.…”

Section: Reverse-adaptive Schedulermentioning

confidence: 99%

“…Existing previous work on MapReduce scheduling involved estimating the resources that needed to be allocated to each job in order to meet its completion goals [6], [7], [8]. This naive estimation worked fine under the assumption that the total amount of resources remained stable over time.…”

Section: Introductionmentioning

confidence: 99%

Adaptive MapReduce Scheduling in Shared Environments

Polo

Becerra

Carrera

et al. 2014

2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing

Self Cite

View full text Add to dashboard Cite

Abstract-In this paper we present a MapReduce task scheduler for shared environments in which MapReduce is executed along with other resource-consuming workloads, such as transactional applications. All workloads may potentially share the same data store, some of them consuming data for analytics purposes while others acting as data generators. This kind of scenario is becoming increasingly important in data centers where improved resource utilization can be achieved through workload consolidation, and is specially challenging due to the interaction between workloads of different nature that compete for limited resources. The proposed scheduler aims to improve resource utilization across machines while observing completion time goals. Unlike other MapReduce schedulers, our approach also takes into account the resource demands for non-MapReduce workloads, and assumes that the amount of resources made available to the MapReduce applications is variable over time. As shown in our experiments, our proposal improves the management of MapReduce jobs in the presence of variable resource availability, increasing the accuracy of the estimations made by the scheduler, thus improving completion time goals without an impact on the fairness of the scheduler.

show abstract

Resource-Aware Adaptive Scheduling for MapReduce Clusters

Cited by 111 publications

References 10 publications

PRISM: Fine-Grained Phase and Resource Information-aware Scheduler for Map-Reduce

PRISM: Fine-Grained Phase and Resource Information-aware Scheduler for Map-Reduce

Optimized Speculative Execution to Improve Performance of MapReduce Jobs on Virtualized Computing Environment

Adaptive MapReduce Scheduling in Shared Environments

Contact Info

Product

Resources

About