BIGhybrid: a simulator for MapReduce applications in hybrid distributed infrastructures validated with the Grid5000 experimental platform

Abstract

Summary Cloud computing has increasingly been used as a platform for running large business and data processing applications. Conversely, Desktop Grids have been successfully employed in a wide range of projects, because they are able to take advantage of a large number of resources provided free of charge by volunteers. A hybrid infrastructure created from the combination of Cloud and Desktop Grids infrastructures can provide a low‐cost and scalable solution for Big Data analysis. Although frameworks like MapReduce have been designed to exploit commodity hardware, their ability to take advantage of a hybrid infrastructure poses significant challenges because of their large resource heterogeneity and high churn rate. In this paper, BIGhybrid is proposed, a simulator for two existing classes of MapReduce runtime environments: BitDew‐MapReduce designed for Desktop Grids and BlobSeer‐Hadoop designed for Cloud computing, where the goal is to carry out accurate simulations of MapReduce executions in a hybrid infrastructure composed of Cloud computing and Desktop Grid resources. This work describes the principles of the simulator and describes the validation of BIGhybrid with the Grid5000 experimental platform. Owing to BIGhybrid, developers can investigate and evaluate new algorithms to enable MapReduce to be executed in hybrid infrastructures. This includes topics such as resource allocation and data splitting. Copyright © 2015 John Wiley & Sons, Ltd.

Fast-Sec: an approach to secure Big Data processing in the cloud

“…Such models allow to analyze the consistency of a proposed security architecture, and figure out the basic adversary model [21]. Other can be applied for distributed data processing infrastructures, such as Flink, possessing more flexibility in providing various data processing functions [5].…”

“…As the systems are independent different data sizes can be manipulated at same time. Previous work has been performed on MapReduce for hybrid environments [5], [28]. Figure 1 illustrates the solution proposed to model a hybrid system which depicts a Global Dis-patcher and Global Aggregator to be used on the infrastructure.…”

“…It is a centralized data storage system that manages policies for split data and distribution in accordance with the needs of each system. The working principle is similar to a publish/subscribe service (PS-S) in which the system obtains data and publishes computing results [5]. PS-S is a mechanism producer/consumer that works like a Queue Message.…”

“…A large volume of data (volume) from different sources that ingress in the cloud with different formats and sources (variety) are processed in real-time (velocity) with high accuracy (veracity) to find competitive information (value) for society. However, data need to be homogenized and coherently handled by the cloud or other infrastructures, like hybrid infrastructures [5]. This infrastructure needs to be secured to guarantee the data privacy and avoid miss usage of the services it offers.…”

Strategies for Big Data Analytics through Lambda Architectures in Volatile Environments

“…BIGhybrid summarizes the main features of a Hybrid MR environment based on the merge of two environments, namely a Cloud (MR-BlobSeer) environment and a Desktop Grid (BitDew-MapReduce) environment (Anjos et al, 2016). The Global Dispatcher located outside the Desktop Grid (DG) as well as of the cloud environment features middleware functionality for handling task assignments and input data from users.…”