ECOS: An efficient task-clustering based cost-effective aware scheduling algorithm for scientific workflows execution on heterogeneous cloud systems

Dong, Minggang; Fan, Lili; Jing, Chao

doi:10.1016/j.jss.2019.110405

Cited by 18 publications

(11 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The comparison was made based on the optimization phase, target, and machine learning approach. From the optimization target perspective, the existing task clustering optimization approaches [9], [10], [17] focus on identifying the imbalance distribution of runtime and dependency of the tasks while maintaining the timeline and budget of the experiment through heuristic approaches, while the other machine-learning optimization approaches [20], [21], [22], [23] are generally improving the scheduler performance by applying machine learning over the scheduling plan and resource allocations. All of the existing studies on task clustering and machine learning optimization approaches are focusing on the execution phase of the workflow lifecycle.…”

Section: Discussionmentioning

confidence: 99%

“…Chen et al [9] proposed an algorithm that clustering the task based on the balance between the task run-time and dependency in order to improve the overall makespan of the workflow execution. Based on Chen et al balance task clustering approach, Dong et al [10] further enhance the algorithm toward cost-effective awareness by clustering the tasks vertically and horizontally with a greedy allocation of the resources in the cloud environment without missing the deadline constraint (part of the QoS in a cloud environment) [16]. Avinash et al [17] assessed the task-dependency of the workflow by calculating the impact factors on the workflow structure tasks that focus on single parent single child relationships and available resources to make the clustering decision.…”

Section: Related Workmentioning

confidence: 99%

“…The situation may become even worst when the scheduling is performed in the large-scale computing systems where both the scheduling and coordination time introduce latency when compared with the execution time. Numerous research studies [9], [10], [11] proposed a task clustering approach to cluster the fine-grained tasks into the coarsegrained task in order to reduce the number of tasks, i.e. reduces the overhead and optimizes the performance of the workflow execution.…”

Section: Introductionmentioning

confidence: 99%

“…Qualitative comparison of related work Optimizing task clustering load balance problem with heuristic approach toward runtime and dependency imbalance Dong et al[10] Execution phase Improve task clustering with cost and time awareness capabilities…”

mentioning

confidence: 99%

See 3 more Smart Citations

Optimizing Workflow Task Clustering Using Reinforcement Learning

et al. 2021

View full text Add to dashboard Cite

Scientific workflows are composed of many fine-grained computational tasks. Generally, a large number of small tasks will slow down the workflow performance due to the scheduling overhead incurs during the execution time. Task clustering is an optimization technique that aggregates multiple small tasks into a large task to reduce the scheduling overhead, and thus it will reduce the overall workflow makespan, i.e. the total execution time taken by the resources to complete the execution of all of the tasks. However, finding an optimal clustering number is a big challenge as it usually requires manual intervention of experienced researchers to define the clustering parameter. In this paper, we proposed the use of reinforcement learning to tackle this problem by automating the discovery of the optimal clustering number for the submitted workflow. First, we model the workflow environment that allows the reinforcement learning agent to interact by determining the cluster number for every round of workflow execution. Then, based on the provenance records after the execution, the workflow environment will analyze the performance data and then determine either a reward or a punishment as the feedback to the reinforcement learning agent. The evaluation experiments are performed using real-world scientific workflow (e.g. Montage in this research), to demonstrate our model capability to identify the optimal cluster number and thus lay the groundwork for the adoption of reinforcement learning in workflow task clustering.

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Related Workmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

mentioning

confidence: 99%

See 2 more Smart Citations

Optimizing Workflow Task Clustering Using Reinforcement Learning

et al. 2021

View full text Add to dashboard Cite

show abstract

“…No fairness in energy consumption and security issues not handled ECOS [32] The cost minimization without comprising the deadline constraint and light-weight complexity algorithm…”

Section: B Contributionmentioning

confidence: 99%

A Predictive Priority-Based Dynamic Resource Provisioning Scheme With Load Balancing in Heterogeneous Cloud Computing

Sohani

Jain

2021

IEEE Access

View full text Add to dashboard Cite

Optimization techniques for task scheduling criteria in IaaS cloud computing atmosphere using nature inspired hybrid spotted hyena optimization algorithm

Natesan¹,

Ali

Pradeep

et al. 2022

Concurrency and Computation

View full text Add to dashboard Cite

Summary Cloud computing has garnered unprecedented growth in recent years in the field of Information Technology. It has emerged as a high‐performance computing option owing to its infrastructure that comprises of heterogeneous collection of autonomous computers and adaptable network architecture. The tasks that are scheduled in an optimized manner for their execution could be classified under NP‐hard problems. Though meta‐heuristic scheduling algorithms emerge as scheduling options, they need to be much more consistent while dealing with the dynamic set up of the cloud environment. In this paper, we had proposed a multi‐objective meta‐heuristic scheduling algorithm namely Quasi Oppositional Genetic Spotted Hyena Optimization (QOGSHO) algorithm that globally optimizes the makespan, resource consumption and SLA violation QoS parameters, thereby improving the performance. The algorithm proposed is an amalgamated product of meta‐heuristic algorithms like Quasi Oppositional Based Learning (QOBL), Spotted Hyena Optimization (SHO), and Genetic Algorithm (GA). The performance efficiency of the proposed QOGSHO algorithm had been compared with various scheduling algorithms using uniform datasets by varying the data instance sizes in a simulated cloud environment. The obtained results clearly justify the task scheduling efficiency of the proposed algorithm with respect to the QoS parameters namely makespan, resource utilization and SLA violation.

show abstract

ECOS: An efficient task-clustering based cost-effective aware scheduling algorithm for scientific workflows execution on heterogeneous cloud systems

Cited by 18 publications

References 25 publications

Optimizing Workflow Task Clustering Using Reinforcement Learning

Optimizing Workflow Task Clustering Using Reinforcement Learning

A Predictive Priority-Based Dynamic Resource Provisioning Scheme With Load Balancing in Heterogeneous Cloud Computing

Optimization techniques for task scheduling criteria in IaaS cloud computing atmosphere using nature inspired hybrid spotted hyena optimization algorithm

Contact Info

Product

Resources

About