iCATS: Scheduling Big Data Workflows in the Cloud Using Cultural Algorithms

Mojab, Seyed Ziae Mousavi; Ebrahimi, Mahdi; Reynolds, Robert G.; Lu, Shiyong

doi:10.1109/bigdataservice.2019.00020

Cited by 6 publications

(2 citation statements)

References 18 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The work of Reference presents an evolutionary algorithm with a fitness function combining time and monetary costs. It is a deadline focused workflow scheduler.…”

Section: Related Workmentioning

confidence: 99%

Optimizing computational resource management for the scientific gateways ecosystems based on the service‐oriented paradigm

et al. 2020

View full text Add to dashboard Cite

Science Gateways provide portals for experiments execution, regardless of the users' computational background. Nowadays its construction and performance need enhancement in terms of resource provision and task scheduling. We present the Modular Distributed Architecture to support the Protein Structure Prediction (MDAPSP), a Service-Oriented Architecture for management and construction of Science Gateways, with resource provisioning on a heterogeneous environment. The Decision Maker, central module of MDAPSP, defines the best computational environment according to experiment parameters. The proof of concept for MDAPSP is presented in WorkflowSim, with two novel schedulers. Our results demonstrate good Quality of Service (QoS), capable of correctly distributing the workload, fair response times, providing load balance, and overall system improvement. The study case relies on PSP algorithms and the Galaxy framework, with monitoring experiments to show the bottlenecks and critical aspects. K E Y W O R D Soptimization, protein structure prediction, science gateways, scientific workflow, service-oriented architecture, WorkflowSim 1 Softw: Pract Exper. 2020;50:899-924. wileyonlinelibrary.com/journal/spe © 2020 John Wiley & Sons, Ltd. 899 900 MARTINS DE OLIVEIRA et al.Another limitation of scientific gateways is the resource provision for its machines, that affect usability and performance. The problem with resource provisioning is a quest for the best configuration, requiring the smallest investment and minimal overheads. In general, inexperienced users cannot make good decisions on what machines to contract. Limited economic resources complicate the scenario.Cloud computing paradigm is among the principal resources available, as it has many advantages: scalability, pay-as-you-go features, on-demand provisioning, and management by third parties. Multiple providers infrastructure and decentralization of resources are expanding and are expected to create impact on diverse areas. 4 Resource providing and management are one of the challenging aspects of this expansion. 5 Despite its success, cloud is not the only option for provisioning. 6 Computer clusters are reliable infrastructures for conducting experiments, mainly High-Performance Computing (HPC) and concurrent algorithms. The advantages of cluster machines are local management, freedom of configuration, reliable storage, and fast network transfers. 3 For those reasons, we investigate the scientific gateway Galaxy, because it is an adaptable system, capable to operate over different infrastructures, including local workstations.Galaxy is a well-known framework for scientific experiments, with workflow management and reproducibility support. Its tools offer practical solutions and addition of new algorithms according to the evolution of the system. The focus is Biomedical science, protein prediction, gene sequencing, and other bioinformatics-related areas. Its advantages are simple interface, reliability, powerful tools, and an active community. 7 However, as any other...

show abstract

“…The work of Reference presents an evolutionary algorithm with a fitness function combining time and monetary costs. It is a deadline focused workflow scheduler.…”

Section: Related Workmentioning

confidence: 99%

Optimizing computational resource management for the scientific gateways ecosystems based on the service‐oriented paradigm

et al. 2020

View full text Add to dashboard Cite

show abstract

“…The scientific application has a huge impact on the DAG size [12]. If the scientific application is simple together less complicated, the workflow size is tiny, or else it is taken as large [13]. The methodology, which performs workflow tasks mapping on the heterogeneous together with distributed resources of a computing system is termed Workflow Scheduling (WFS).…”

mentioning

confidence: 99%

A Multi-Objective Based Scheduling Framework for Effective Resource Utilization in Cloud Computing

Reddy

Karri

2023

IEEE Access

View full text Add to dashboard Cite

Cloud computing is a promising platform for running massive workflow applications based on a pay-per-use model. In cloud computing, the reduction of energy consumption and providing security to workflow scheduling are the key research areas. The primary focus of the existing algorithms, viz., particle swarm optimization (PSO), crow Search optimization (CSO) and Other non-metaheuristic algorithms like Round Robbin(RR), SJF, Min-Min, Min-Max etc., is based on the execution time and cost of the workflow applications as a budget constraint. However, these algorithms failed to adequately determine energy consumption, resource utilization, and security in workflow scheduling. To address this issue, a multi-objective scheduling framework is proposed. In this paper, the framework performs dynamic workflow scheduling using universal unique identification-Blake (UUID-Blake), Manhattan Distance-Partition around algorithm (MD-PAM), Linear Scaling-Crow Search Optimization (LS-CSO), Anova-Recurrent Neural Network. The implementation of this framework was achieved in three phases (Phase 1, Phase 2, and Phase 3). Phase 1 is about user registration and authentication using UUID-Blake, which enhances security by allowing legitimate users into the cloud environment. Phase 2 deals with clustering and resource monitoring using MD-PAM and A-RNN, to reduce makespan the similar tasks are clustered using task length and maximize the resource utilization by predicting the resource availability. Phase 3 deals with the scheduling of dynamic workflows using LS-CSO by selecting suitable virtual machines. We have considered the heterogeneous computing scheduling problem (HCSP) and grid workload archive (GWA)-T-12 Bitbrains datasets for comparing our proposed framework with existing works. Based on the result analysis, the proposed LS-SCO outperformed when compared with the algorithms CSO,PSO and RR has achieved better performance. INDEX TERMSCrow swarm optimization, Linear scaling, Manhattan distance, Partitioning around medoid, Recurrent neural network, Workflow scheduling. I. INTRODUCTION C Loud computing is transforming into a highperformance computing environment with enormous computational resources such as storage, networking, databases, and applications. These resources are allocated to users from the collective resource pool with reduced management or interaction [1]. The three major services delivered by the cloud technology include Software-as-a-Service (SaaS), Platform-asa-Service (PaaS), along with Infrastructure-as-a-Service (IaaS) [2] out of which the most prominent model is IaaS. This model provides pre-configured virtual machines (VMs) to the user from the cloud framework [3] [4]. An infinite quantity of computational resources and storage facilities can be attained by using VMs [5].With the increased demand for such cloud services, the number of users is expanding day by day [6]. Because of this, the unpredictable resource usage of VMs has increased, due to which the servers of cloud data centers get either over-utilized or un...

show abstract

Applications of Cultural Algorithms in Different Branches of Science

Jalili¹

2022

Engineering Optimization: Methods and Applications

View full text Add to dashboard Cite

iCATS: Scheduling Big Data Workflows in the Cloud Using Cultural Algorithms

Cited by 6 publications

References 18 publications

Optimizing computational resource management for the scientific gateways ecosystems based on the service‐oriented paradigm

Optimizing computational resource management for the scientific gateways ecosystems based on the service‐oriented paradigm

A Multi-Objective Based Scheduling Framework for Effective Resource Utilization in Cloud Computing

Applications of Cultural Algorithms in Different Branches of Science

Contact Info

Product

Resources

About