Enhancement of Hadoop Clusters with Virtualization Using the Capacity Scheduler

Raj, Aparna; Kaur, Kamaldeep; Dutta, Uddipan; Sandeep, V Venkat; Rao, Shrisha

doi:10.1109/icsem.2012.15

Cited by 16 publications

(10 citation statements)

References 1 publication

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It increases performance by a factor of 2 on 200 VMs on EC2. Similarly, there are several approaches focused on enhancing the task scheduler of the Hadoop framework to reduce power consumption or network cost . Kim et al defined an intercloud as a federated environment of public cloud and private clusters and proposed a task scheduler to enhance performance.…”

Section: Related Workmentioning

confidence: 99%

Enhancing network I/o performance for a virtualized Hadoop cluster

Jeong

Choi

2016

Concurrency and Computation

View full text Add to dashboard Cite

Summary A MapReduce programming model is proposed to process big data using Hadoop, one of the major cloud computing frameworks. With the increasing adoption of cloud computing, running a Hadoop framework on a virtualized cluster is a compelling approach to reducing costs and increasing efficiency. In this paper, we measure the performance of a virtualized network and analyze the impact of network performance on Hadoop workloads running on a virtualized cluster. Then, we propose a virtualized network I/O architecture as a novel optimization for a virtualized Hadoop cluster for a public/private cloud provider. The proposed network architecture combines traditional network configurations and achieves better performance for Hadoop workloads. We also show a better way to utilize the rack awareness feature of the Hadoop framework in the proposed computing environment. The evaluation demonstrates that the proposed network architecture and mechanisms improve performance by up to 4.1 times compared with a bridge network architecture. This novel architecture can even virtually match the performance of the expensive, hardware‐based single root I/O virtualization network architecture.

show abstract

Section: Related Workmentioning

confidence: 99%

Enhancing network I/o performance for a virtualized Hadoop cluster

Jeong

Choi

2016

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…Most of work focuses on optimizing the scheduling policies to meet different requirements in a centralized task scheduler. The Hadoop default schedulers include the Capacity Scheduler (CS) [19], the Fair Scheduler (FS) [20] and the Hadoop On Demand (HOD) Scheduler (HS) [21]. Each of them has a different design goal: the CS aims at offering resource sharing to multiple tenants with the individual capacity and performance SLA; the FS divides resources fairly among job pools to ensure that the jobs get an equal share of resources over time; the HS relies on the Torque resource manager to allocate nodes, and allows users to easily setup Hadoop by provisioning tasks and HDFS instances on the nodes.…”

Section: Related Workmentioning

confidence: 99%

Overcoming Hadoop Scaling Limitations through Distributed Task Execution

Wang

Liu

Sadooghi

et al. 2015

2015 IEEE International Conference on Cluster Computing

View full text Add to dashboard Cite

Data driven programming models like MapReduce have gained the popularity in large-scale data processing. Although great efforts through the Hadoop implementation and framework decoupling (e.g. YARN, Mesos) have allowed Hadoop to scale to tens of thousands of commodity cluster processors, the centralized designs of the resource manager, task scheduler and metadata management of HDFS file system adversely affect Hadoop's scalability to tomorrow's extreme-scale data centers. This paper aims to address the YARN scaling issues through a distributed task execution framework, MATRIX, which was originally designed to schedule the executions of data-intensive scientific applications of many-task computing on supercomputers. We propose to leverage the distributed design wisdoms of MATRIX to schedule arbitrary data processing applications in cloud. We compare MATRIX with YARN in processing typical Hadoop workloads, such as WordCount, TeraSort, Grep and RandomWriter, and the Ligand application in Bioinformatics on the Amazon Cloud. Experimental results show that MATRIX outperforms YARN by 1.27X for the typical workloads, and by 2.04X for the real application. We also run and simulate MATRIX with fine-grained sub-second workloads. With the simulation results giving the efficiency of 86.8% at 64K cores for the 150ms workload, we show that MATRIX has the potential to enable Hadoop to scale to extreme-scale data centers for fine-grained workloads.

show abstract

“…In addition, it improves resources sharing among the queues in order to fairly distribute the available cluster capacity among users rather than among the submitted jobs (Fair Scheduler). The capacity scheduler relies only on memory capacity allocation to optimize the execution time and the throughput of the data-intensive jobs within Hadoop large clusters [14].…”

Section: Capacity Schedulermentioning

confidence: 99%

ATLAS: An AdapTive faiLure-Aware Scheduler for Hadoop

Soualhia

Khomh

Tahar

2015

2015 IEEE 34th International Performance Computing and Communications Conference (IPCCC)

View full text Add to dashboard Cite

Hadoop has become the de facto standard for processing large data in today's cloud environment. The performance of Hadoop in the cloud has a direct impact on many important applications ranging from web analytic, web indexing, image and document processing to high-performance scientific computing. However, because of the scale, complexity and dynamic nature of the cloud, failures are common and these failures often impact the performance of jobs running in Hadoop. Although Hadoop possesses built-in failure detection and recovery mechanisms, several scheduled jobs still fail because of unforeseen events in the cloud environment. A single task failure can cause the failure of the whole job and unpredictable job running times. In this paper, we propose ATLAS (AdapTive faiLure-Aware Scheduler), a new scheduler for Hadoop that can adapt its scheduling decisions to events occurring in the cloud environment. Using statistical models, ATLAS predicts task failures and adjusts its scheduling decisions on the fly to reduce task failure occurrences. We implement ATLAS in the Hadoop framework of Amazon Elastic MapReduce (EMR) and perform a case study to compare its performance with those of the FIFO, Fair and Capacity schedulers. Results show that ATLAS can reduce the percentage of failed jobs by up to 28% and the percentage of failed tasks by up to 39%, and the total execution time of jobs by 10 minutes on average. ATLAS also reduces CPU and memory usages.

show abstract

Enhancement of Hadoop Clusters with Virtualization Using the Capacity Scheduler

Cited by 16 publications

References 1 publication

Enhancing network I/o performance for a virtualized Hadoop cluster

Enhancing network I/o performance for a virtualized Hadoop cluster

Overcoming Hadoop Scaling Limitations through Distributed Task Execution

ATLAS: An AdapTive faiLure-Aware Scheduler for Hadoop

Contact Info

Product

Resources

About