Cost-efficient dynamic scheduling of big data applications in apache spark on cloud

Islam, Muhammed Tawfiqul; Srirama, Satish Narayana; Karunasekera, Shanika; Buyya, Rajkumar

doi:10.1016/j.jss.2019.110515

Cited by 44 publications

(20 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Moreover, these resources are obtained back to cluster. 21,22 The resources allocation is an on-demand service, which may lead to service loss. It can be prevented by assigning resources to divide the parameters other than partitioning the whole cloud model.…”

Section: Introductionmentioning

confidence: 99%

“…Thus, the allotment of resources is present in the class of resources management. 18 An effective job scheduling technique is devised in Reference [21] that minimizes the cost of utilizing cloud-based Apache Spark clusters whereas improving job performance. Here, a Genetic Algorithm (GA) is devised in Reference [8] to predict resource utilization in the upcoming time slot, which involves memory utilization, central processing unit (CPU), and energy constraints.…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Exponentially‐spider monkey optimization based allocation of resource in cloud

Kalpana

Prabhu²,

Polepally

et al. 2021

Int J of Intelligent Sys

View full text Add to dashboard Cite

The increasing desire for distributed computing systems has attracted huge interest in memory and computing resources. The cloud provides on-demand access to provide a flexible allocation of resources for reliable services. Therefore, there should be a provision in which resources are accessible to request users to satisfy user needs. In classical techniques, the allocation of resources by satisfying power and Quality-of-Services is a challenging aspect. This paper devises a novel technique for optimal resource allocation, namely, Exponentially Spider Monkey Optimization (E-SMO). Here, the proposed E-SMO is devised by combining Exponential Weighted Moving Average and Spider Monkey Optimization (SMO). Besides, the fitness function is newly devised considering resource utilization and resource cost. After that, the cloud resources employ a switching strategy to reduce power consumption to prevent the switching of redundant servers. For optimal switching, the proposed E-SMO is utilized with other fitness factors that compute the number of applications assigned in the physical machine using the switching state is in OFF condition. Thus, the server switching model is incorporated to activate or deactivate the server when not in use for effective resources utilization. The proposed E-SMO algorithm outperformed other methods with the maximal resource

show abstract

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Exponentially‐spider monkey optimization based allocation of resource in cloud

Kalpana

Prabhu²,

Polepally

et al. 2021

Int J of Intelligent Sys

View full text Add to dashboard Cite

show abstract

“…In CDC, the majority of computing resources are modelled as Virtual machine (VM) that is mapped to physical machine (PM). The performance is a key measure for CDC (Islam et al, 2020;ElKafhali & Salah, 2017). The optimization performance can be effectively achieved by monitoring the usage of computing resources.…”

Section: Introductionmentioning

confidence: 99%

Optimal Switching and Load Distribution Strategy for Energy Minimization and Performance Optimization of Cloud Data Center

Poobalan

Sangeetha

Shanthakumar³

2021

JER is an international, peer-reviewed journal that publishes f

View full text Add to dashboard Cite

Cloud computing is a promising computing technology utilized in every stage of the business. The cloud offers different services to cloud users from anytime to anywhere, and it is attained with different parameters, like load optimization, resource optimization. Due to the increase in data center, energy consumption has become a major issue in green data centers. The majority of data centers are function using peak load with huge scales. Thus, it is essential for carrying out energy saving in cloud data centers. This paper designed an energy-saving method using fat tree. The proposed techniques optimize the load at different zones of data center and user in the cloud platform. Here, the distribution of load in cloud data centers is performed using Taylor-based Manta Ray Foraging Optimization (Taylor-MRFO), which is an integration of Manta Ray Foraging Optimization (MRFO) and Taylor series. The method utilized different objectives that involve power, load, latency, and bandwidth. With the load distribution, the switching of cloud data center to the desired mode is performed using Actor critic neural network (ACNN). Thus, the dual strategy leads to performance optimization in cloud infrastructure and also in consolidating parallel workload in data centers more effectively. The proposed Taylor-MRFO+ACNN outperformed other methods with minimal energy of 0.553, minimal load of 0.363, and minimal fitness of 0.437, respectively.

show abstract

“…Cloud computing has become a backbone for the IT infrastructure of different industries and organizations, viz., education, weather forecasting, hedge funding, e‐commerce, and big data solution 3,4 . Kochan et al 5 used a cloud computing based framework in the domain of hospital supply chain to enhance the performance of demand and supply of healthcare items 3,6,7 …”

Section: Introductionmentioning

confidence: 99%

HEART: Unrelated parallel machines problem with precedence constraints for task scheduling in cloud computing using heuristic and meta‐heuristic algorithms

et al. 2020

View full text Add to dashboard Cite

Summary Cloud computing is becoming a profitable technology because of it offers cost‐effective IT solutions globally. A well‐designed task scheduling algorithm ensures the optimal utilization of clouds resources and reducing execution time dynamically. This research article deals with the task scheduling of inter‐dependent subtasks on unrelated parallel computing machines in a cloud computing environment. This article considers two variants of the problem‐based on two different objective function values. The first variant considers the minimization of the total completion time objective function while the second variant considers the minimization of the makespan objective function. Heuristic and meta‐heuristic (HEART) based algorithms are proposed to solve the task scheduling problems. These algorithms utilize the property of list scheduling algorithm of unrelated parallel machine scheduling problem. A mixed integer linear programming (MILP) formulation has been provided for the two variants of the problem. The optimal solution is obtained by solving MILP formulation using A Mathematical Programming Language (AMPL) software. Extensive numerical experiments have been performed to evaluate the performance of proposed algorithms. The solutions obtained by the proposed algorithms are found to out‐perform the existing algorithms. The proposed algorithms can be used by cloud computing service providers (CCSPs) for enhancing their resources utilization to reduce their operating cost.

show abstract

Cost-efficient dynamic scheduling of big data applications in apache spark on cloud

Cited by 44 publications

References 6 publications

Exponentially‐spider monkey optimization based allocation of resource in cloud

Exponentially‐spider monkey optimization based allocation of resource in cloud

Optimal Switching and Load Distribution Strategy for Energy Minimization and Performance Optimization of Cloud Data Center

HEART: Unrelated parallel machines problem with precedence constraints for task scheduling in cloud computing using heuristic and meta‐heuristic algorithms

Contact Info

Product

Resources

About