Optimal Tradeoff between Energy Consumption and Response Time in Large-Scale MapReduce Clusters

Abstract

Abstract-The increasing growth of the size of the digital databases has given rise to the need for the development of infrastructures, such as large scale data centers and computational clusters, which are capable of storing and processing very large volumes of data. To date, most clusters have been designed for performance. Due to non-linear speed-ups that are common to typical applications, performance maximization involves the decision of the number of the nodes to process a specific (intensive) task, as opposed to the utilization of the full cluster. In addition, energy consumption has recently attracted significant attention, given that the cost to operate a cluster may well exceed its acquisition cost. This issue calls for judicious use of resources as well. The aim of this study is to present a method that achieves the optimal tradeoff between energy consumption and response time in distributed clusters, such as MapReduce clusters. To this end, we propose an algorithm that derives the fraction of the nodes that minimizes the energy consumption without sacrificing performance (in terms of response time) more than a user-defined threshold. Moreover, we present a generic and configurable framework to describe performance and energy consumption as a function of the nodes used; our framework can accommodate the widely spread MapReducelike parallel executions in a straightforward manner. The evaluation results show that our methodology can lead to significant energy savings with acceptable performance penalty in many realistic situations.

Optimizing MapReduce for energy efficiency

“…To estimate the energy consumption of MapReduce applications, a few simple energy models based on power and time have been used in various empirical studies. These performance and energy models are based on the number of nodes, average power, and execution time of tasks/jobs for evaluating the relative improvements in energy efficiency by using different solutions …”

“…These performance and energy models are based on the number of nodes, average power, and execution time of tasks/jobs for evaluating the relative improvements in energy efficiency by using different solutions. 6,24,25 Li et al 26 recognized that the energy consumed by the CPU, disk, and network are the key contributors to the energy consumption by a server. Hence, they proposed a model to predict the energy consumption based on (1) the number of instructions in a job (which represents CPU usage), (2) the size of input and output files (to represent disk I/O usage), and (3) the total number of bytes transferred during the shuffle phase (to represent network usage).…”

Task Scheduling in Big Data Platforms: A Systematic Literature Review

“…Paraskevopoulos et al [110] propose a strategy to schedule the tasks for Hadoop, while balancing between energy consumption and response time. Their proposed strategy can help identify the nodes in a cluster that can satisfy the constraint of less energy in a reasonable response time.…”