Hybrid gradient descent spider monkey optimization (HGDSMO) algorithm for efficient resource scheduling for big data processing in heterogenous environment

Abstract

Big Data constructed based on the advancement of distributed computing and virtualization is considered as the current emerging trends in Data Analytics. It is used for supporting potential utilization of computing resources focusing on, on-demand services and resource scalability. In particular, resource scheduling is considered as the process of resource distribution through an effective decision making process with the objective of facilitating required tasks over time. The incorporation of heterogeneous computing resources by the Big Data consumers also permits the option of reducing energy usage and enhanced resource efficiency. Further, optimal scheduling of resources is considered as an NP hard problem due to the dynamic characteristics of the resources and fluctuating users’ demand. In this paper, a Hybrid Gradient Descent Spider Monkey Optimization (HGDSMO) algorithm is proposed to efficient resource scheduling by handling the issues and challenges in the Hadoop heterogenous environment. The proposed HGDSMO algorithm uses the Gradient Descentand foraging and social behavior of the spider monkey optimization algorithm involved in the objective of effective resource allocation. It is designed as the efficient task scheduling approach that balances the load of the cloud by allocating them to appropriate VMs depending on their requirements. It is also proposed as a dynamic resource management scheme for efficiently allocating the cloud resources for effective execution of clients’ tasks. The simulation results of the proposed HGDSMO algorithm confirmed to be potent in throughput, load balancing and makespan compared to the baseline hybrid meta-heuristic resource allocation algorithms used for investigation.

Security of Big Data over IoT Environment by Integration of Deep Learning and Optimization

“…The proposed HGDSMO algorithm takes inspiration from the foraging and social behaviour of spider monkeys to attain the same aim of effective resource allocation. [15] Daissaoui, et al (2020) researched a survey on IoT & big data analytics for smart buildings. The goal of the digital transformation procedures has been to increase productivity, safety, and execution quality, as well as to promote sustainable development, teamwork, and solutions for the sustainable smart city mainstream digital advancements are revealing new tendencies in the integration of information technology into the construction sector, which is undergoing radical change as a result.…”

SMOSA: Spider monkey optimization‐based scheduling algorithm for heterogeneous Hadoop

“…Through the test of 25 benchmark problems and various statistical tests, they pointed out that SMO is a competitive optimization algorithm. 29 For the task scheduling problem, Seethalakshmi et al 30 designed a hybrid gradient descent spider monkey optimization (HGDSMO) algorithm to solve the task scheduling problem in a heterogeneous Hadoop environment. The algorithm obtained the best task allocation plan at a faster speed by combining the gradient descent local search method and the spider monkey foraging social behavior for global optimization.…”

New efficient Hadoop scheduler: Generalized particle swarm optimization and simulated annealing‐dominant resource fairness

“…The author in References 9,26 proposed a solution for minimizing the execution time with data locality in a heterogeneous environment. Another heterogeneous environment load balancing solution is discussed in Reference 10 with the help of the hybrid gradient descent spider monkey optimization (HGDSMO) algorithm. The greedy approach is followed to assign the task in the heterogeneous environment 17 and once slots are allotted, the update in slots is handled by fuzzy logic for the efficient use of resource utilization.…”