Performance Optimization for Short MapReduce Job Execution in Hadoop

Yan, Jinshuang; Yang, Xiaoliang; Gu, Rong; Yuan, Chunfeng; Huang, Yihua

doi:10.1109/cgc.2012.40

Cited by 18 publications

(8 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…[7] proposed parallel computing framework for solving the problem of data intensive applications. In this paper reduce the time cost of initialization and termination stage, pull model is replaced by push model with task assignment mechanism and message communication mechanism between task tracker and job.…”

Section: Literature Surveymentioning

confidence: 99%

A Survey on Performance and Security of Hadoop

2017

IJSR

View full text Add to dashboard Cite

Hadoop Framework is used to process big data in parallel fashion. A big data is not only big in size but also it is in different format, different size and with different speeds. To process big data relational database management system is not a suitable one. The hadoop is a most popular framework to process big data. Hadoop framework architecture has many components like name node data, data node, job tracker and task tracker. The performance of hadoop is dependent on how these components execute. The challenge in Hadoop framework is to reduce processing time of job, but these challenges are depend upon various factors like scheduling, performance of map reduce after data encryption, resource allocation, and data encryption. Proposed research is focused on how to overcome these challenge of scheduling, resource allocation, and security. Hadoop data security is also a proposed research area i.e. to find a best suitable encryption algorithm which encrypts hadoop data without affecting hadoop performance.

show abstract

Section: Literature Surveymentioning

confidence: 99%

A Survey on Performance and Security of Hadoop

2017

IJSR

View full text Add to dashboard Cite

show abstract

“…A group of researchers have proposed optimization approaches for a particular type of jobs such as short jobs and query based jobs . Jahani et al .…”

Section: Related Workmentioning

confidence: 99%

“…A group of researchers have proposed optimization approaches for a particular type of jobs such as short jobs and query based jobs [46][47][48][49]. Jahani et al proposed the MANIMAL model [46], which automatically analyzes a Hadoop program using a static analyzer tool for optimization.…”

Section: Related Workmentioning

confidence: 99%

Optimizing hadoop parameter settings with gene expression programming guided PSO

Khan

Huang

et al. 2016

Concurrency and Computation

View full text Add to dashboard Cite

SUMMARYHadoop MapReduce has become a major computing technology in support of big data analytics. The Hadoop framework has over 190 configuration parameters, and some of them can have a significant effect on the performance of a Hadoop job. Manually tuning the optimum or near optimum values of these parameters is a challenging task and also a time consuming process. This paper optimizes the performance of Hadoop by automatically tuning its configuration parameter settings. The proposed work first employs gene expression programming technique to build an objective function based on historical job running records, which represents a correlation among the Hadoop configuration parameters. It then employs particle swarm optimization technique, which makes use of the objective function to search for optimal or near optimal parameter settings. Experimental results show that the proposed work enhances the performance of Hadoop significantly compared with the default settings. Moreover, it outperforms both rule-of-thumb settings and the Starfish model in Hadoop performance optimization.

show abstract

“…So the execution time of large jobs will be increase. Jinshuang Yan et al [10] also propose a job scheduling, which has advantages in the time cost during the initial phase of a job, and the task assignment because the push-model replaces the pull-model which is a task assignment mechanism. But, it can not perform well for large jobs, because it is designed for small jobs.…”

Section: Related Workmentioning

confidence: 99%

“…If the size of task cannot be found that exactly matches the size, the closest match is used. T a s k 7 T a s k 8 T a s k 9 T a s k 10 T h e c u r r e n tly r u n n in g ta s k…”

Section: The Real-time Prediction Model Of Jobsmentioning

confidence: 99%