Deriving Job Completion Reliability and Job Energy Consumption for a General MapReduce Infrastructure from Single-Job Perspective

Lin, Jia-Chun; Leu, Fang−Yie; Lee, Ming-Chang; Chen, Ying-ping

doi:10.1109/waina.2013.10

Cited by 3 publications

(1 citation statement)

References 12 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This has been investigated in two different academic papers. On the one hand, (Lin et al, 2013) analytically models the Job Completion Reliability in MapReduce frameworks when the cold-standby redundancy scheme is applied and nodes may crash with a given probability. On the other hand, (Sangroya et al 2012) does not use any simulations as the other work, but defines a proper dependability benchmarking framework for studying the fault-tolerance guarantees of MapReduce implementations by applying fault injection.…”

Section: Academic Solutions To Enhance Fault-tolerancementioning

confidence: 99%

Recent Developments on Security and Reliability in Large-Scale Data Processing with MapReduce

Esposito

Ficco

2016

International Journal of Data Warehousing and Mining

View full text Add to dashboard Cite

The demand to access to a large volume of data, distributed across hundreds or thousands of machines, has opened new opportunities in commerce, science, and computing applications. MapReduce is a paradigm that offers a programming model and an associated implementation for processing massive datasets in a parallel fashion, by using non-dedicated distributed computing hardware. It has been successfully adopted in several academic and industrial projects for Big Data Analytics. However, since such analytics is increasingly demanded within the context of mission-critical applications, security and reliability in MapReduce frameworks are strongly required in order to manage sensible information, and to obtain the right answer at the right time. In this paper, the authors present the main implementation of the MapReduce programming paradigm, provided by Apache with the name of Hadoop. They illustrate the security and reliability concerns in the context of a large-scale data processing infrastructure. They review the available solutions, and their limitations to support security and reliability within the context MapReduce frameworks. The authors conclude by describing the undergoing evolution of such solutions, and the possible issues for improvements, which could be challenging research opportunities for academic researchers.

show abstract

Section: Academic Solutions To Enhance Fault-tolerancementioning

confidence: 99%

Recent Developments on Security and Reliability in Large-Scale Data Processing with MapReduce

Esposito

Ficco

2016

International Journal of Data Warehousing and Mining

View full text Add to dashboard Cite

show abstract

Impact of MapReduce Task Re-execution Policy on Job Completion Reliability and Job Completion Time

Lin¹,

Leu

Chen

et al. 2014

2014 IEEE 28th International Conference on Advanced Information Networking and Applications

View full text Add to dashboard Cite

Analyzing job completion reliability and job energy consumption for a general MapReduce infrastructure

Lin

Leu

Chen

2013

Journal of High Speed Networks

View full text Add to dashboard Cite

Recently, MapReduce has been a popular distributed programming framework, which divides a job into map tasks and reduce tasks and executes these tasks in parallel over a large-scale MapReduce cluster to speed up job execution. Generally, the cluster is a master-slave infrastructure. To prevent jobs from being interrupted due to node failure, current MapReduce implementations, such as Hadoop, adopt a task-reexecution policy on the slave side, i.e., when a slave node due to failure cannot complete a task, this task will be reassigned to another available slave for reexecution. However, on the master side by default, no redundancy scheme is provided. Since this type of infrastructure has been worldwide adopted, we call it the general MapReduce infrastructure (GMI). To achieve a more reliable and energy-efficient working environment, understanding the impact of GMI on its job completion reliability (JCR) and job energy consumption (JEC) is required. In this paper, we base on a Poisson distribution to analyze GMI's JCR from a single-job perspective. After that, we accordingly derive the corresponding JEC. Through the analytical results, MapReduce managers can comprehend how GMI behaves and how their MapReduce can be improved so as to achieve a more reliable and energy-efficient MapReduce environment.

show abstract

Deriving Job Completion Reliability and Job Energy Consumption for a General MapReduce Infrastructure from Single-Job Perspective

Cited by 3 publications

References 12 publications

Recent Developments on Security and Reliability in Large-Scale Data Processing with MapReduce

Recent Developments on Security and Reliability in Large-Scale Data Processing with MapReduce

Impact of MapReduce Task Re-execution Policy on Job Completion Reliability and Job Completion Time

Analyzing job completion reliability and job energy consumption for a general MapReduce infrastructure

Contact Info

Product

Resources

About