Analysis and Improvement of Makespan and Utilization for MapReduce

Yin, Li; Chen, Lin; Ren, Fengyuan

doi:10.1109/hpcc.and.euc.2013.69

2013 IEEE 10th International Conference on High Performance Computing and Communications &Amp; 2013 IEEE International Conferen 2013

DOI: 10.1109/hpcc.and.euc.2013.69

|View full text |Cite

Analysis and Improvement of Makespan and Utilization for MapReduce

Li Yin

Lin Chen

Fengyuan Ren

Help me understand this report

Search citation statements

Order By: Relevance

Paper Sections

Select...

Citation Types

Supporting

Mentioning

Contrasting

Year Published

2014

Publication Types

Select...

Other1

Relationship

Self Cite0

Independent1

Authors

Journals

Cited by 1 publication

References 10 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

Scalability Analysis and Improvement of Hadoop Virtual Cluster with Cost Consideration

Jiang

et al. 2014

2014 IEEE 7th International Conference on Cloud Computing

View full text Add to dashboard Cite

With the rapid development of big data and cloud computing, big data analytics as a service in the cloud is becoming increasingly popular. More and more individuals and organizations tend to rent virtual cluster to store and analyze data rather than building their own data centers. However, in virtualization environment, whether scaling out using a cluster with more nodes to process big data is better than scaling up by adding more resources to the original virtual machines (VMs) in cluster is not clear. In this paper, we study the scalability performance issues of hadoop virtual cluster with cost consideration. We first present the design and implementation of VirtualMR platform which can provide users with scalable hadoop virtual cluster services for the MapReduce based big data analytics. Then we run a series of hadoop benchmarks and real parallel machine learning algorithms to evaluate the scalability performance, including scale-up method and scale-out method. Finally, we integrate our platform with resource monitoring module and propose a system tuner. By analyzing the monitored data, we dynamically adjust the parameters of hadoop framework and virtual machine configuration to improve resource utilization and reduce rent cost. Experimental results show that the scaleup method outperforms the scale-out method for CPU-bound applications, and it is opposite for I/O-bound applications. The results also verify the efficiency of system tuner to increase resource utilization and reduce rent cost.

show abstract

Scalability Analysis and Improvement of Hadoop Virtual Cluster with Cost Consideration

Jiang

et al. 2014

2014 IEEE 7th International Conference on Cloud Computing

View full text Add to dashboard Cite

show abstract

scite is a Brooklyn-based organization that helps researchers better discover and understand research articles through Smart Citations–citations that display the context of the citation and describe whether the article provides supporting or contrasting evidence. scite is used by students and researchers from around the world and is funded in part by the National Science Foundation and the National Institute on Drug Abuse of the National Institutes of Health.

Contact Info

customersupport@researchsolutions.com

10624 S. Eastern Ave., Ste. A-614

Henderson, NV 89052, USA

This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Blog Terms and Conditions API Terms Privacy Policy Contact Cookie Preferences Do Not Sell or Share My Personal Information

Made with 💙 for researchers

Part of the Research Solutions Family.

Analysis and Improvement of Makespan and Utilization for MapReduce

Cited by 1 publication

References 10 publications

Scalability Analysis and Improvement of Hadoop Virtual Cluster with Cost Consideration

Scalability Analysis and Improvement of Hadoop Virtual Cluster with Cost Consideration

Contact Info

Product

Resources

About