Estimating runtime of a job in Hadoop MapReduce

Abstract

Nowadays, with the emergence and use of new systems, we face a massive amount of data. Due to the volume, velocity, and variety of these big data, managing, maintaining, and processing them require special infrastructures. One of the famous open-source frameworks is Apache Hadoop [1]. It is a scalable and reliable framework for storage and process big data. Hadoop divides the big input data into fixed-size pieces; stores and processes these split of data on a cluster of machines. By default, each split copies three times and transfer to different machines to manage errors and fault tolerance. Hadoop stores its data in a distributed file system called HDFS. MapReduce is designed to work with HDFS. MapReduce is the programming model that allows Hadoop to efficiently process large amounts of data in the cluster's nodes. [1-5]. Since one of Hadoop's most important tasks is managing jobs and resources, better management will be done if estimation and prediction the runtime of a job do precisely. Also, because of limited critical resources like CPU, I/O, and memory, this issue is important in many aspects like efficient scheduling, better energy consumption, bottleneck detection, and resource management [3-5].

Uniform Machine Scheduling with Predictions

“…This work considers using ML oracle to predict job sizes. Recent works (Amiri and Mohammad-Khanli 2017;Peyravi and Moeini 2020;Yamashiro and Nonaka 2021) have shown that job sizes are highly predictable in many scenarios, e.g., cloud, clusters, and factories. In addition, when the prediction is accurate, we have the 2-relaxed decision procedure guaranteeing a near-optimal makespan, and when the prediction goes arbitrarily bad, the existing O(log m)-competitive algorithm can bound the performance.…”

An Analytic solution for the Hadoop Configuration Combinatorial Puzzle based on General Factorial Design

“…Zhu et al [16] proposed BestConfig, which uses the divide-anddiverge sampling method and the recursive-bound-and-search method for parameter tuning of general systems with resource constraints. Peyravi et al [17] reported the runtime efficiency of a MapReduce job by considering three categories of parameters that have higher impact on the runtime. They have modeled the runtime efficiency during each phase of the Hadoop execution pipeline using a weighing system based of job history.…”