Evaluating MapReduce on Virtual Machines: The Hadoop Case

Ibrahim, Shadi; Jin, Hai; Lu, Lu; Li, Qi; Wu, Song; Shi, Xuanhua

doi:10.1007/978-3-642-10665-1_47

Cited by 95 publications

(33 citation statements)

References 8 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Ibrahim et al in [26], analyze Hadoop execution time on virtual machines. Stewart in [27] compares the performance of several data query languages.…”

Section: Related Workmentioning

confidence: 99%

Performance characterization and analysis for Hadoop K-means iteration

Issa

2016

J Cloud Comp

View full text Add to dashboard Cite

The rapid growth in the demand for cloud computing data presents a performance challenge for both software and hardware architects. It is important to analyze and characterize the data processing performance for a given cloud cluster and to evaluate the performance bottlenecks in a cloud cluster that contribute to higher or lower computing processing time. In this paper, we implement a detailed performance analysis and characterization for Hadoop K-means iterations by scaling different processor micro-architecture parameters and comparing performance using Intel and AMD processors. This leads to the analysis of the underlying hardware in a cloud cluster servers to enable optimization of software and hardware to achieve maximum performance possible. We also propose a performance estimation model that estimates performance for Hadoop K-means iterations by modeling different processor micro-architecture parameters. The model is verified to predict performance with less than 5 % error margin relative to a measured baseline.

show abstract

“…Ibrahim et al in [26], analyze Hadoop execution time on virtual machines. Stewart in [27] compares the performance of several data query languages.…”

Section: Related Workmentioning

confidence: 99%

Performance characterization and analysis for Hadoop K-means iteration

Issa

2016

J Cloud Comp

View full text Add to dashboard Cite

show abstract

“…Hadoop performance in virtualized environments can benefit from such mechanisms. Previous work [22], has shown that VMs are suitable for executing data intensive Hadoop applications through use of sort and wordcount benchmarks. The work by Jian et.…”

Section: Related Workmentioning

confidence: 99%

Performance and energy efficiency of big data applications in cloud environments: A Hadoop case study

Feller

Ramakrishnan

Morin

2015

Journal of Parallel and Distributed Computing

View full text Add to dashboard Cite

The exponential growth of scientific and business data has resulted in the evolution of the cloud computing environments and the MapReduce parallel programming model. The focus of cloud computing is increased utilization and power savings through consolidation while MapReduce enables large scale data analysis.Hadoop, an open source implementation of MapReduce has gained popularity in the last few years. In this paper, we evaluate Hadoop performance in both the traditional model of collocated data and compute services as well as consider the impact of separating out the services. The separation of data and compute services provides more flexibility in environments where data locality might not have a considerable impact such as virtualized environments and clusters with advanced networks. In this paper, we also conduct an energy efficiency evaluation of Hadoop on physical and virtual clusters in different configurations. Our extensive evaluation shows that: (1) coexisting virtual machines on servers decrease the disk throughput; (2) performance on physical clusters is significantly better than on virtual clusters; (3) performance degradation due to separation of the services depends on the data to compute ratio; (4) application completion progress correlates with the power consumption and power consumption is heavily application specific. Finally, we present a discussion on the implications of using cloud environments for big data analyses.

show abstract

“…It is not uncommon to have some of the nodes having more running processes than other nodes. The situation becomes more significant when virtualization is employed [17]. Since each of the virtualized machines (i.e., VMs) has to compete for resources from the physical machines, spontaneous congestions can happen frequently [15].…”

Section: Jobsmentioning

confidence: 99%

An Eye on the Elephant in the Wild: A Performance Evaluation of Hadoop’s Schedulers Under Failures

Ibrahim

Phuong

Antoniu

2015

Adaptive Resource Management and Scheduling for Cloud Computing

Self Cite

View full text Add to dashboard Cite

Abstract. Large-scale data analysis has increasingly come to rely on MapReduce and its open-source implementation Hadoop. Recently, Hadoop has not only been used for running single batch jobs but it has also been optimized to simultaneously support the execution of multiple jobs belonging to multiple concurrent users. Several schedulers (i.e., Fifo, Fair, and Capacity schedulers) have been proposed to optimize locality executions of tasks but do not consider failures, although, evidence in the literature shows that faults do occur and can probably result in performance problems. In this paper, we have designed a set of experiments to evaluate the performance of Hadoop under failure when applying several schedulers (i.e., explore the conflict between job scheduling, exposing locality executions, and failures). Our results reveal several drawbacks of current Hadoop's mechanism in prioritizing failed tasks. By trying to launch failed tasks as soon as possible regardless of locality, it significantly increases the execution time of jobs with failed tasks, due to two reasons: 1) available resources might not be freed up as quickly as expected and 2) failed tasks might be re-executed on machines with no data on it, introducing extra cost for data transferring through network, which is normally the most scarce resource in today's data-centers. Our preliminary study with Hadoop not only helps us to understand the interplay between fault-tolerance and job scheduling, but also offers useful insights into optimizing the current schedulers to be more efficient in case of failures.

show abstract

Evaluating MapReduce on Virtual Machines: The Hadoop Case

Cited by 95 publications

References 8 publications

Performance characterization and analysis for Hadoop K-means iteration

Performance characterization and analysis for Hadoop K-means iteration

Performance and energy efficiency of big data applications in cloud environments: A Hadoop case study

An Eye on the Elephant in the Wild: A Performance Evaluation of Hadoop’s Schedulers Under Failures

Contact Info

Product

Resources

About