Estimation Accuracy on Execution Time of Run-Time Tasks in a Heterogeneous Distributed Environment

Liu, Qi; Cai, Weidong; Jin, Dandan; Shen, Jian; Fu, Zhangjie; Liu, Xiaodong; Linge, Nigel

doi:10.3390/s16091386

Cited by 17 publications

(7 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Liu et al [9] proposed a two-phase regression (TPR) method to predict the finishing time of each job precisely. Detailed data of each job had made with a detailed analysis [1,11] report.…”

Section: Background and Related Workmentioning

confidence: 99%

Estimating runtime of a job in Hadoop MapReduce

Peyravi

Moeini

2020

J Big Data

View full text Add to dashboard Cite

Nowadays, with the emergence and use of new systems, we face a massive amount of data. Due to the volume, velocity, and variety of these big data, managing, maintaining, and processing them require special infrastructures. One of the famous open-source frameworks is Apache Hadoop [1]. It is a scalable and reliable framework for storage and process big data. Hadoop divides the big input data into fixed-size pieces; stores and processes these split of data on a cluster of machines. By default, each split copies three times and transfer to different machines to manage errors and fault tolerance. Hadoop stores its data in a distributed file system called HDFS. MapReduce is designed to work with HDFS. MapReduce is the programming model that allows Hadoop to efficiently process large amounts of data in the cluster's nodes. [1-5]. Since one of Hadoop's most important tasks is managing jobs and resources, better management will be done if estimation and prediction the runtime of a job do precisely. Also, because of limited critical resources like CPU, I/O, and memory, this issue is important in many aspects like efficient scheduling, better energy consumption, bottleneck detection, and resource management [3-5].

show abstract

“…Liu et al [9] proposed a two-phase regression (TPR) method to predict the finishing time of each job precisely. Detailed data of each job had made with a detailed analysis [1,11] report.…”

Section: Background and Related Workmentioning

confidence: 99%

Estimating runtime of a job in Hadoop MapReduce

Peyravi

Moeini

2020

J Big Data

View full text Add to dashboard Cite

show abstract

“…The Analytical methods used mathematical models like machine learning, regression methods to predict the performance measures. Liu et al [13] proposed a finish time prediction method for distributed tasks. The real data is collected by modifying MapReduce and regression is applied to predict the accurate finish time.…”

Section: Related Workmentioning

confidence: 99%

Untitled

2020

IAJIT

View full text Add to dashboard Cite

Cloud computing is on-demand network access model which provides dynamic resource provisioning, selection and scheduling. The performance of these techniques extensively depends on the prediction of various factors e.g., task execution time, resource trust value etc., As the accuracy of prediction model absolutely depends on the input data that are fed into the network, Selection of suitable inputs also plays vital role in predicting the appropriate value. Based on predicted value, Scheduler can choose the suitable resource and perform scheduling for efficient resource utilization and reduced makespan estimates. However, precise prediction of execution time is difficult in cloud environment due to heterogeneous nature of resources and varying input data. As each task has different characteristic and execution criteria, the environment must be intelligent enough to select the suitable resource. To solve these issues, an Artificial Neural Network (ANN) based prediction model is proposed to predict the execution time of tasks. First, input parameters are identified and selected through Interpretive Structural Modeling (ISM) approach. Second, a prediction model is proposed for predicting the task execution time for varying number of inputs. Third, the proposed model is validated and provides 21.72% reduction in mean relative error compared to other state-of-the-art methods.

show abstract

“…However, MCP still contains a lot of pitfalls, and it can only be used in MR, not including MRV2 that has adopted a newly resource management framework. Liu et al proposed a new model by collecting real-time data, which has achieved higher accuracy [24]. A dynamic strategy has been proposed based on an exponential smoothing model for each phase [25].…”

Section: Related Workmentioning

confidence: 99%

“…While theoretically infinite computing resources can be provided in a cloud, the unreasonable increment of mappers/reducers cannot achieve process efficiency, and may waste more storage to complete. Many optimization schemes have been proposed [8][9][10][11][12][13][14][15][16][17][18][19][20][21][22][23][24][25].…”

Section: Introductionmentioning

confidence: 99%

An Adaptively Speculative Execution Strategy Based on Real-Time Resource Awareness in a Multi-Job Heterogeneous Environment

2017

KSII TIIS

View full text Add to dashboard Cite

MapReduce (MRV1), a popular programming model, proposed by Google, has been well used to process large datasets in Hadoop, an open source cloud platform. Its new version MapReduce 2.0 (MRV2) developed along with the emerging of Yarn has achieved obvious improvement over MRV1. However, MRV2 suffers from long finishing time on certain types of jobs. Speculative Execution (SE) has been presented as an approach to the problem above by backing up those delayed jobs from low-performance machines to higher ones. In this paper, an adaptive SE strategy (ASE) is presented in Hadoop-2.6.0. Experiment results have depicted that the ASE duplicates tasks according to real-time resources usage among work nodes in a cloud. In addition, the performance of MRV2 is largely improved using the ASE strategy on job execution time and resource consumption, whether in a multi-job environment.

show abstract

Estimation Accuracy on Execution Time of Run-Time Tasks in a Heterogeneous Distributed Environment

Cited by 17 publications

References 33 publications

Estimating runtime of a job in Hadoop MapReduce

Estimating runtime of a job in Hadoop MapReduce

Untitled

An Adaptively Speculative Execution Strategy Based on Real-Time Resource Awareness in a Multi-Job Heterogeneous Environment

Contact Info

Product

Resources

About