Energy-aware scheduling for spark job based on deep reinforcement learning in cloud

Li, Hongjian; Lu, Liang; Shi, Wenhu; Tan, Gangfan; Luo, Hao

doi:10.1007/s00607-023-01171-z

Cited by 4 publications

(3 citation statements)

References 30 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The fitness function has proposed to arrange the assignment in cloud environment for optimizing the energy consumed and time of execution, the fitness function has used for finding nearest optimal solution. Fitness functions for makespan and energy consumption shown in the Equation (8).…”

Section: Step 3: Fitness Functionmentioning

confidence: 99%

“…6,7 The energy consuming data centers is anticipated to increase by 53% by 2020 to reach 140 billion kWh, which is equivalent to 50 power plants' annual output and will cost American business $13 billion in electricity costs and almost 100 million metric tons carbon pollution annually. 8 As a result of the cloud's increased operational costs brought on by excessive energy use and the knowledge of global warming, numerous efforts have been made to reduce the energy consumption of data centers. 9,10 The energy efficiency is increased by increasing the framework of hardware components utilizing effective power distribution options, creating strategies for effective air handling, 11 cooling measures.…”

Section: Introductionmentioning

confidence: 99%

“…According to Natural Resources Defense Council report, the data centers in the United States utilized 91 billion kWh of energy at 2013 that is parallel to 34 sizable coal‐fired plants 6,7 . The energy consuming data centers is anticipated to increase by 53% by 2020 to reach 140 billion kWh, which is equivalent to 50 power plants' annual output and will cost American business $13 billion in electricity costs and almost 100 million metric tons carbon pollution annually 8 . As a result of the cloud's increased operational costs brought on by excessive energy use and the knowledge of global warming, numerous efforts have been made to reduce the energy consumption of data centers 9,10 .…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Hybrid Pelican and Archimedes optimization algorithm fostered energy aware task scheduling in heterogeneous virtualized cloud computing

S.,

Shanmugam,

et al. 2023

Trans Emerging Tel Tech

View full text Add to dashboard Cite

Energy reduction is a key issue of virtualized cloud computing (CC) schemes, because it provide significant benefits, like lower operating costs, improving system effectiveness, securing the environment. Energy‐efficient task scheduling is a feasible mode to attain these objectives. Nevertheless, it is difficult to match cloud resources to user requests in a way that improves performance while meeting a user‐defined deadline for energy consumption reduction. Therefore, hybrid Pelican and Archimedes optimization algorithm fostered energy‐aware task scheduling in heterogeneous virtualized cloud computing (TS‐HVCC‐Hyb‐POA‐AOA) is proposed in this article. A hybrid Pelican and Archimedes optimization algorithm (Hyb‐POA‐AOA) is used to lessen the task duration and the consumption of power in the cloud. For evaluation, the cloud environment uses the provided environment hybrid Pelican and Archimedes optimization algorithm for execute a few common workloads at the simulated data center. The performance metrics, like makespan right skewed analysis, makespan left skewed analysis, energy consumption right skewed analysis, energy consumption left skewed analysis, availability and resource utilization is considered. The performance of the proposed TS‐HVCC‐Hyb‐POA‐AOA method provides 23.69%, 29.50%, and 36.78% higher resource utilization; 38.23%, 31.35%, and 26.19% lower consumption of energy left skewed analysis and 34.52%, 30.28%, and 23.54% lower makespan for right skewed analysis compared with existing methods such as; task scheduling in heterogeneous cloud environment with gray wolf optimization algorithm (TS‐HVCC‐GWOA), hybrid whale optimization approach and differential evolution optimization for multiple objective virtual machine programming at cloud services (TS‐HVCC‐WOA), energy and performance planning in TasterkHeterek virtualized cloud computing using particle swarm optimization algorithm (TS‐HVCC‐PSOA).

show abstract

Section: Step 3: Fitness Functionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Hybrid Pelican and Archimedes optimization algorithm fostered energy aware task scheduling in heterogeneous virtualized cloud computing

S.,

Shanmugam,

et al. 2023

Trans Emerging Tel Tech

View full text Add to dashboard Cite

show abstract

Deep reinforcement learning-based methods for resource scheduling in cloud computing: a review and future directions

Zhou,

Tian,

Buyya

et al. 2024

Artif Intell Rev

View full text Add to dashboard Cite

With the acceleration of the Internet in Web 2.0, Cloud computing is a new paradigm to offer dynamic, reliable and elastic computing services. Efficient scheduling of resources or optimal allocation of requests is one of the prominent issues in emerging Cloud computing. Considering the growing complexity of Cloud computing, future Cloud systems will require more effective resource management methods. In some complex scenarios with difficulties in directly evaluating the performance of scheduling solutions, classic algorithms (such as heuristics and meta-heuristics) will fail to obtain an effective scheme. Deep reinforcement learning (DRL) is a novel method to solve scheduling problems. Due to the combination of deep learning and reinforcement learning (RL), DRL has achieved considerable performance in current studies. To focus on this direction and analyze the application prospect of DRL in Cloud scheduling, we provide a comprehensive review for DRL-based methods in resource scheduling of Cloud computing. Through the theoretical formulation of scheduling and analysis of RL frameworks, we discuss the advantages of DRL-based methods in Cloud scheduling. We also highlight different challenges and discuss the future directions existing in the DRL-based Cloud scheduling.

show abstract

Energy-latency trade-off analysis for scientific workflow in cloud environments: The role of processor utilization ratio and mean grey wolf optimizer

Khaleel,

Safran,

Alfarhood

et al. 2024

Engineering Science and Technology, an International Journal

View full text Add to dashboard Cite

Energy-aware scheduling for spark job based on deep reinforcement learning in cloud

Cited by 4 publications

References 30 publications

Hybrid Pelican and Archimedes optimization algorithm fostered energy aware task scheduling in heterogeneous virtualized cloud computing

Hybrid Pelican and Archimedes optimization algorithm fostered energy aware task scheduling in heterogeneous virtualized cloud computing

Deep reinforcement learning-based methods for resource scheduling in cloud computing: a review and future directions

Energy-latency trade-off analysis for scientific workflow in cloud environments: The role of processor utilization ratio and mean grey wolf optimizer

Contact Info

Product

Resources

About