Resource Management and Scheduling for Big Data Applications in Cloud Computing Environments

Abstract

This chapter presents software architectures of the big data processing platforms. It also provides in-depth knowledge on resource management techniques involved while deploying big data processing systems in the cloud environment. It starts from the very basics and gradually introduce the core components of resource management which are divided into multiple layers. It covers the state-of-art practices and researches done in SLA-based resource management with a specific focus on the job scheduling mechanisms.

An Innovative Industry 4.0 Cloud Data Transfer Method for an Automated Waste Collection System

“…A huge amount of data are available in manufacturing systems, waste management systems, etc., but the data and information should be properly collected, encoded and transmitted to the cloud application in order to be processed and to allow a real-time monitoring of equipment and systems. The major challenges are represented by: data acquisition from smart sensors, actuators and PLCs; data conversion; data security and privacy; data encoding, encryption and decryption [7][8][9][10].Software architectures of the big data processing platforms are analyzed in the literature, including solutions for job scheduling for big data applications [11], solutions which offer a flexible co-programming architecture able to support the life cycle of time-critical cloud native applications [12], mobility-driven cloud-fog-edge collaborative real-time framework solution, which has IoT, Edge, Fog and Cloud layers and which exploits the mobility dynamics of the moving agent [13].Several data encoding and encryption solutions for cloud applications are presented in the literature [14][15][16][17][18][19][20][21][22], but without including detailed case studies to present identified problems, propose solutions, and implement results.The main problem is the identification of the right solution that will ensure the data transmission and processing in a secure way, quickly, efficiently and with low costs. The proposed solution presented in Chapter 2 can be used both in smart factories and for smart city applications including automated waste collection systems.Waste management comprises prevention, collection (metal, paper, cardboard, aluminum cans plastic), removal, recycling, reuse, and safe disposal of toxic and dangerous waste.…”

“…Software architectures of the big data processing platforms are analyzed in the literature, including solutions for job scheduling for big data applications [11], solutions which offer a flexible co-programming architecture able to support the life cycle of time-critical cloud native applications [12], mobility-driven cloud-fog-edge collaborative real-time framework solution, which has IoT, Edge, Fog and Cloud layers and which exploits the mobility dynamics of the moving agent [13].…”