Dynamic energy efficient data placement and cluster reconfiguration algorithm for MapReduce framework

Maheshwari, Nitesh; Nanduri, Radheshyam; Varma, Vasudeva

doi:10.1016/j.future.2011.07.001

Cited by 93 publications

(36 citation statements)

References 27 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Data replication algorithms can be classified into two groups: static replication [15] and dynamic replication algorithms [16][17]. In a static replication model, the replication strategy is predetermined and well defined.…”

Section: Data Sets Replicas Placements Technologymentioning

confidence: 99%

Combination Replicas Placements Strategy for Data sets from Cost-effective View in the Cloud

Wu¹

2017

IJCIS

View full text Add to dashboard Cite

In the cloud storage system, data sets replicas technology can efficiently enhance data availability and thereby increase the system reliability by replicating commonly used data sets in geographically different data centers. Most current approaches largely focus on system performance improvement by placing replicas for an independent data set, omitting the generation relationship among data sets. Furthermore, cost is an important element in deciding replicas number and their stored places, which can cause great financial burden for cloud clients because the cost for replicas storage and consistency maintenance may lead to high overhead with the number of new replicas increased in a pay-as-you-go paradigm. In this paper, we propose a combination strategy of real-replicas and pseudo-replicas (by computation from its provenance) from cost-effective view in order to achieve the minimum data set management cost, not only for the independent data sets but also for related data sets with generation relationships. We first define cost models that fit into the cloud computing paradigm, including data sets storage, computation and transfer costs, and then develop a new data set management cost model, helping to achieve a multi-criteria optimization of data set management. After that, a minimum cost benchmarking approach for the best trade-off between real-replicas and pseudo-replicas is proposed once decision to add a replica has been made. Then, a more practical and reasonable genetic algorithm as an alternative procedure for generating optimal or nearoptimal solution is given in order to identify the suitable replicas storage places. Finally, we present simulations setups and results that provide a first validation of our strategy. Both the theoretical analysis and simulations conducted on general (random) data sets as well as specific real world applications have shown efficiency and effectiveness of the improved system brought by the proposed strategy in cloud computing environment.

show abstract

Section: Data Sets Replicas Placements Technologymentioning

confidence: 99%

Combination Replicas Placements Strategy for Data sets from Cost-effective View in the Cloud

Wu¹

2017

IJCIS

View full text Add to dashboard Cite

show abstract

“…As proposed in [12], the insertion of a power controller node with the only task of turning on or off the required nodes as needed reducing idle power consumption, renders a negligible increase in provisioning costs (figure 5) for a relatively positive increase in energy efficiency in the case of user and microserver-grade processor clusters, as seen in Figure 6 where ECI values (green, with legend ECI) are related to the previously studied case (blue, with legend Std).…”

Section: ) Energy Controller Insertionmentioning

confidence: 99%

A Study on the Energetic Viability of Single Board Computers for Cloud Computing Scenarios

Verdugo

Salvachúa

Huecas

2017

Proceedings XIII Jornadas De Ingenieria Telematica - JITEL2017

View full text Add to dashboard Cite

Abstract-The following document explores the viability of the usage of consumer-grade, ARM-based single board computers as a power saving alternative to the traditional monolithic x64-full-server based approach. By taking advantage of several capabilities provided by such devices, such as low cost, low power consumption and low on-time, the authors finally propose a scalable, energy-efficient, ARMbased cloud infrastructure. To that end, we start analyzing the current offerings in terms of capabilities, net cost, processing power and power consumption, comparing them with the relevant server-oriented offerings. We subsequently explore the adequacy of several metrics to model on-budget raw data processing, considering full-system wattage under nominal usage conditions. The low initial investment and long-term affordability of this approach results in quite a relevant case of application to Edge Cloud computing scenarios.

show abstract

“…The above approaches focus on removing cluster nodes, but the automation for the scaling operations is barely noted. Aiming to achieve automation, Maheshwari et al proposed a dynamic data placement and cluster reconfiguration algorithm to turn off or on cluster nodes based on the average cluster utilization rate [26]. However, it requires data blocks stored on one node to be transferred to other nodes before node removal.…”

Section: Hadoop For Geospatial Data Processingmentioning

confidence: 99%

Automatic Scaling Hadoop in the Cloud for Efficient Process of Big Geospatial Data

Yang

Liu

et al. 2016

IJGI

View full text Add to dashboard Cite

Efficient processing of big geospatial data is crucial for tackling global and regional challenges such as climate change and natural disasters, but it is challenging not only due to the massive data volume but also due to the intrinsic complexity and high dimensions of the geospatial datasets. While traditional computing infrastructure does not scale well with the rapidly increasing data volume, Hadoop has attracted increasing attention in geoscience communities for handling big geospatial data. Recently, many studies were carried out to investigate adopting Hadoop for processing big geospatial data, but how to adjust the computing resources to efficiently handle the dynamic geoprocessing workload was barely explored. To bridge this gap, we propose a novel framework to automatically scale the Hadoop cluster in the cloud environment to allocate the right amount of computing resources based on the dynamic geoprocessing workload. The framework and auto-scaling algorithms are introduced, and a prototype system was developed to demonstrate the feasibility and efficiency of the proposed scaling mechanism using Digital Elevation Model (DEM) interpolation as an example. Experimental results show that this auto-scaling framework could (1) significantly reduce the computing resource utilization (by 80% in our example) while delivering similar performance as a full-powered cluster; and (2) effectively handle the spike processing workload by automatically increasing the computing resources to ensure the processing is finished within an acceptable time. Such an auto-scaling approach provides a valuable reference to optimize the performance of geospatial applications to address data-and computational-intensity challenges in GIScience in a more cost-efficient manner.

show abstract

Dynamic energy efficient data placement and cluster reconfiguration algorithm for MapReduce framework

Cited by 93 publications

References 27 publications

Combination Replicas Placements Strategy for Data sets from Cost-effective View in the Cloud

Combination Replicas Placements Strategy for Data sets from Cost-effective View in the Cloud

A Study on the Energetic Viability of Single Board Computers for Cloud Computing Scenarios

Automatic Scaling Hadoop in the Cloud for Efficient Process of Big Geospatial Data

Contact Info

Product

Resources

About