Modeling and predicting execution time of scientific workflows in the Grid using radial basis function neural network

Nadeem, Farrukh; Alghazzawi, Daniyal; Mashat, Abdulfattah S.; Fakeeh, Khalid A.; AL-Ghamdi, Abdullah S. AL-Malaise; Hagras, Hani

doi:10.1007/s10586-017-1018-x

Cited by 28 publications

(25 citation statements)

References 53 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…e majority of algorithms rely on the estimation of tasks execution time in appropriate resources to produce an accurate schedule plan. Meanwhile, the works on task runtime estimation in scienti c work ows are limited including the latest works by Pham et al (Pham et al 2017) and Nadeem et al (Nadeem et al 2017) that used machine learning techniques while previously, a work on scienti c work ows pro ling and characterization by Juve et al (Juve et al 2013) that produced a synthetic work ows generator is being used by the majority of works on work ow scheduling. e future task runtime prediction techniques must be able to address dynamic workloads in multi-tenant platforms that are continuously arriving in resemblance to stream data processing.…”

Section: Fast and Reliable Task Runtime Estimation In Near Real-time mentioning

confidence: 99%

Multiple Workflows Scheduling in Multi-tenant Distributed Systems

2020

View full text Add to dashboard Cite

e work ow is a general notion representing the automated processes along with the ow of data. e automation ensures the processes being executed in the order. erefore, this feature a racts users from various background to build the work ow. However, the computational requirements are enormous and investing for a dedicated infrastructure for these work ows is not always feasible. To cater to the broader needs, multi-tenant platforms for executing work ows were began to be built. In this paper, we identify the problems and challenges in the multiple work ows scheduling that adhere to the platforms. We present a detailed taxonomy from the existing solutions on scheduling and resource provisioning aspects followed by the survey of relevant works in this area. We open up the problems and challenges to shove up the research on multiple work ows scheduling in multi-tenant distributed systems.

show abstract

Section: Fast and Reliable Task Runtime Estimation In Near Real-time mentioning

confidence: 99%

Multiple Workflows Scheduling in Multi-tenant Distributed Systems

2020

View full text Add to dashboard Cite

show abstract

“…Workflow execution time can be effectively modeled using different attributes reflecting workflow structure and execution environment. Nadeem and Fahringer [8] and Nadeem et al [9] presented a comprehensive framework defining attributes that can effectively model workflow execution time in the Grid. We used their framework to parameterize workflow performance in terms of workflow attributes.…”

Section: Parameterizing Performance Of Scientific Workflow Applicmentioning

confidence: 99%

“…selected scheduling algorithm [4], selected Grid-sites [10] and their states, etc. For the sake of better understanding the proposed approach, we are reproducing the detailed attributes from [8] and [9] as below.…”

Section: Parameterizing Performance Of Scientific Workflow Applicmentioning

confidence: 99%

Using Machine Learning Ensemble Methods to Predict Execution Time of e-Science Workflows in Heterogeneous Distributed Systems

et al. 2019

Self Cite

View full text Add to dashboard Cite

Effective planning and optimized execution of the e-Science workflows in distributed systems, such as the Grid, need predictions of execution times of the workflows. However, predicting the execution times of e-Science workflows in heterogeneous distributed systems is a challenging job due to the complex structure of workflows, variations due to input problem-sizes, and heterogeneous and dynamic nature of the shared resources. To this end, we propose two novel workflow execution time-prediction methods based on the machine learning ensemble models. In this paper, we showcase our approach for two different real Grid environments. Our approach can effectively predict the execution time of the scientific workflow applications in the Grid for various problem sizes, Grid sites, and runtime environments. We characterized the workflow performance in the Grid using the attributes that define structure of workflow as well as the execution environment. Contrary to common ensembles, our ensemble systems employed three strong learners, which balance the weaknesses of each other by their strengths to model the workflow execution times. The proposed methods have been thoroughly evaluated for three real-world e-science workflow applications. The experimental results demonstrated that our proposed multi-model ensemble models can significantly decrease the prediction error (by 50%, on average) as compared with methods based on the radial basis function neural network, local learning, and performance templates. The proposed methods can also be applied with similar effectiveness and without any major modification for other heterogeneous distributed environments, such as the Cloud. INDEX TERMS Machine learning ensemble systems, performance modeling of e-science workflows, distributed execution of e-science workflows, execution time prediction of e-science workflows.

show abstract

“…Event calculus [96] [ [99][100][101] MPI [97] Parallel programming Query distribution [87,98] Stream processing…”

Section: Computation Modelmentioning

confidence: 99%

“…Reasoning techniques [71] Planning [72] Computational architecture Infrastructure [86][87][88] Scheduling techniques [89] Workflow…”

mentioning

confidence: 99%

Survey of Scientific Programming Techniques for the Management of Data-Intensive Engineering Environments

Álvarez-Rodríguez

Alor-Hernández

Mejia-Miranda

2018

Scientific Programming

View full text Add to dashboard Cite

The present paper introduces and reviews existing technology and research works in the field of scientific programming methods and techniques in data-intensive engineering environments. More specifically, this survey aims to collect those relevant approaches that have faced the challenge of delivering more advanced and intelligent methods taking advantage of the existing large datasets. Although existing tools and techniques have demonstrated their ability to manage complex engineering processes for the development and operation of safety-critical systems, there is an emerging need to know how existing computational science methods will behave to manage large amounts of data. That is why, authors review both existing open issues in the context of engineering with special focus on scientific programming techniques and hybrid approaches. 1193 journal papers have been found as the representative in these areas screening 935 to finally make a full review of 122. Afterwards, a comprehensive mapping between techniques and engineering and nonengineering domains has been conducted to classify and perform a meta-analysis of the current state of the art. As the main result of this work, a set of 10 challenges for future data-intensive engineering environments have been outlined.

show abstract

Modeling and predicting execution time of scientific workflows in the Grid using radial basis function neural network

Cited by 28 publications

References 53 publications

Multiple Workflows Scheduling in Multi-tenant Distributed Systems

Multiple Workflows Scheduling in Multi-tenant Distributed Systems

Using Machine Learning Ensemble Methods to Predict Execution Time of e-Science Workflows in Heterogeneous Distributed Systems

Survey of Scientific Programming Techniques for the Management of Data-Intensive Engineering Environments

Contact Info

Product

Resources

About