To compress or not to compress - compute vs. IO tradeoffs for mapreduce energy efficiency

Chen, Yanpei; Ganapathi, Archana; Katz, Randy H.

doi:10.1145/1851290.1851296

Cited by 81 publications

(50 citation statements)

References 4 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…It joins others in which the focus is to understand the energy consumption of compression techniques in different scenarios [9], [10]. To the best of our knowledge, this work is the first to study the impact of data deduplication on storage system energy consumption.…”

Section: Introductionmentioning

confidence: 88%

See 1 more Smart Citation

Assessing data deduplication trade-offs from an energy and performance perspective

Costa

Al-Kiswany

Lopes

et al. 2011

2011 International Green Computing Conference and Workshops

View full text Add to dashboard Cite

Abstract-The energy costs of running computer systems are a growing concern: for large data centers, recent estimates put these costs higher than the cost of hardware itself. As a consequence, energy efficiency has become a pervasive theme for designing, deploying, and operating computer systems. This paper evaluates the energy trade-offs brought by data deduplication in distributed storage systems. Depending on the workload, deduplication can enable a lower storage footprint, reduce the I/O pressure on the storage system, and reduce network traffic, at the cost of increased computational overhead. From an energy perspective, data deduplication enables a trade-off between the energy consumed for additional computation and the energy saved by lower storage and network load. The main point our experiments and model bring home is the following: while for non energy-proportional machines performance-and energy-centric optimizations have break-even points that are relatively close, for the newer generation of energy proportional machines the break-even points are significantly different. An important consequence of this difference is that, with newer systems, there are higher energy inefficiencies when the system is optimized for performance.

show abstract

Section: Introductionmentioning

confidence: 88%

“…Recently, Chen et al [9] and Kothiyal et al [10] have investigated the trade-off of compressing or not data in the context of MapReduce applications and data centers, respectively. Similar to this work, they concluded that compression is not always the best choice in terms of energy consumption, as it depends on the workload.…”

Section: B Energy Optimized Systemsmentioning

confidence: 99%

Assessing data deduplication trade-offs from an energy and performance perspective

Costa

Al-Kiswany

Lopes

et al. 2011

2011 International Green Computing Conference and Workshops

View full text Add to dashboard Cite

show abstract

“…In the Hadoop compression algorithms, Andre Wenas [2] used compression as follow: GZIP, LZJB and ZLE for data warehouse and his results shows the best performance on ZLE. Yanpei Chen's research [4] tried to select compress or not compress map reduce output file for reducing power consumption. His results shows that it decreases energy consumption more than 50%.Bhavin J. Mathiya [5] use more compression algorithm as follows: DE-FLATE, LZ4, BZIP and GZIP with word-count benchmark both map output and reduce output.…”

Section: Related Workmentioning

confidence: 99%

Applying compression algorithms on hadoop cluster implementing through apache tez and hadoop mapreduce

Lydia¹,

Rao²

2018

IJET

View full text Add to dashboard Cite

The latest and famous subject all over the cloud research area is Big Data; its main appearances are volume, velocity and variety. The characteristics are difficult to manage through traditional software and their various available methodologies. To manage the data which is occurring from various domains of big data are handled through Hadoop, which is open framework software which is mainly developed to provide solutions. Handling of big data analytics is done through Hadoop Map Reduce framework and it is the key engine of hadoop cluster and it is extensively used in these days. It uses batch processing system. Apache developed an engine named "Tez", which supports interactive query system and it won't writes any temporary data into the Hadoop Distributed File System(HDFS).The paper mainly focuses on performance juxtaposition of MapReduce and TeZ, performance of these two engines are examined through the compression of input files and map output files. To compare two engines we used Bzip compression algorithm for the input files and snappy for the map out files. Word Count and Terasort gauge are used on our experiments. For the Word Count gauge, the results shown that Tez engine has better execution time than Hadoop MapReduce engine for the both compressed and non-compressed data. It has reduced the execution time nearly 39% comparing to the execution time of the Hadoop MapReduce engine. Correspondingly for the terasort gauge, the Tez engine has higher execution time than Hadoop MapReduce engine.

show abstract

“…Consequently, this maximizes the number of idle servers that can be deactivated to save energy. Chen et al [5] analyze how MapReduce parameters affect energy efficiency and discuss the computation versus I/O tradeoffs when using data compression in MapReduce clusters in terms of energy efficiency [4]. Chen et al [3] present the Berkeley Energy Efficient MapReduce (BEEMR), an energy efficient MapReduce workload manager motivated by empirical analysis of real-life MapReduce with Interactive Analysis (MIA) traces at Facebook.…”

Section: Related Workmentioning

confidence: 99%

“…As such large-scale deployments become a distinctive characteristic of cloud infrastructures, energyefficient MapReduce is nowadays an essential concern in data-centers. Several studies have explored power saving in Hadoop clusters, through various techniques [2,4].…”

Section: Introductionmentioning

confidence: 99%

Towards Efficient Power Management in MapReduce: Investigation of CPU-Frequencies Scaling on Power Efficiency in Hadoop

Ibrahim

Moise²,

Chihoub³

et al. 2014

Lecture Notes in Computer Science

View full text Add to dashboard Cite

Abstract. With increasingly inexpensive cloud storage and increasingly powerful cloud processing, the cloud has rapidly become the environment to store and analyze data. Most of the large-scale data computations in the cloud heavily rely on the MapReduce paradigm and its Hadoop implementation. Nevertheless, this exponential growth in popularity has significantly impacted power consumption in cloud infrastructures. In this paper, we focus on MapReduce and we investigate the impact of dynamically scaling the frequency of compute nodes on the performance and energy consumption of a Hadoop cluster. To this end, a series of experiments are conducted to explore the implications of Dynamic Voltage Frequency scaling (DVFS) settings on power consumption in Hadoopclusters. By adapting existing DVFS governors (i.e., performance, powersave, ondemand, conservative and userspace) in the Hadoop cluster, we observe significant variation in performance and power consumption of the cluster with different applications when applying these governors: the different DVFS settings are only sub-optimal for different MapReduce applications. Furthermore, our results reveal that the current CPU governors do not exactly reflect their design goal and may even become ineffective to manage the power consumption in Hadoop clusters. This study aims at providing more clear understanding of the interplay between performance and power management in Hadoop cluster and therefore offers useful insight into designing power-aware techniques for Hadoop systems.

show abstract

To compress or not to compress - compute vs. IO tradeoffs for mapreduce energy efficiency

Cited by 81 publications

References 4 publications

Assessing data deduplication trade-offs from an energy and performance perspective

Assessing data deduplication trade-offs from an energy and performance perspective

Applying compression algorithms on hadoop cluster implementing through apache tez and hadoop mapreduce

Towards Efficient Power Management in MapReduce: Investigation of CPU-Frequencies Scaling on Power Efficiency in Hadoop

Contact Info

Product

Resources

About