LEEN: Locality/Fairness-Aware Key Partitioning for MapReduce in the Cloud

Ibrahim, Shadi; Jin, Hai; Lu, Lu; Wu, Song; He, Bingsheng; Li, Qi

doi:10.1109/cloudcom.2010.25

Cited by 138 publications

(64 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…To find the optimal way to schedule shuffling, we need to (i) choose the nodes where reduce tasks will run; (ii) figure out how to distribute intermediate data to reduce tasks. Ibrahim et al proposed a heuristic algorithm to balance the amount of shuffled data and fair distribution of intermediate data on reduce nodes [23]. However, they implicitly assume the network connections between map tasks and reduce tasks are homogeneous and thus only consider the size of shuffled data.…”

Section: ) Shuffling and Scheduling Of Reduce Tasksmentioning

confidence: 99%

Improving MapReduce Performance in Heterogeneous Network Environments and Resource Utilization

Guo

Fox

2012

2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (Ccgrid 2012)

View full text Add to dashboard Cite

Abstract-In MapReduce, map and reduce tasks are assigned to map and reduce slots hosted by worker nodes. Usually the numbers of map and reduce slots are carefully chosen to gain optimal resource usage. We found resource utilization is inefficient when there are not enough tasks to fill all task slots as the resources "reserved" for idle slots are just wasted. We propose resource stealing which enables running tasks to steal the unutilized resources and return them when new tasks are assigned. It exploits the opportunistic use of the otherwise wasted resources to improve overall resource utilization and reduce job execution time. Besides, our practical use of Hadoop shows the current mechanism adopted to trigger speculative execution creates many unnecessary speculative tasks that are killed soon after creation as the original tasks complete earlier. To alleviate the issue, we propose Benefit Aware Speculative Execution which predicts the benefit of launching new speculative tasks and greatly eliminates unnecessary runs of speculative tasks. Finally, MapReduce is mainly optimized for homogeneous environments and its inefficiency in heterogeneous network environments has been observed in our experiments.We investigate network heterogeneity aware scheduling of both map and reduce tasks. Overall, our goal is to enhance Hadoop to cope with significant system heterogeneity and improve resource utilization.

show abstract

Section: ) Shuffling and Scheduling Of Reduce Tasksmentioning

confidence: 99%

Improving MapReduce Performance in Heterogeneous Network Environments and Resource Utilization

Guo

Fox

2012

2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (Ccgrid 2012)

View full text Add to dashboard Cite

show abstract

“…Recently, some studies have been presented for solving the load balancing problem in MapReduce [3]- [5]. In [3], authors proposed a system, called Scarlett, which adopts a replication policy based on access patterns of files.…”

Section: Related Workmentioning

confidence: 99%

“…The load balancing problem is known as NP-hard [2]. There have been some research works for load balancing in MapReduce, but they still have some scalability problems [3]- [5].…”

Section: Introductionmentioning

confidence: 99%

Skew-Tolerant Key Distribution for Load Balancing in MapReduce

Son

Choi

Chung

2012

IEICE Trans. Inf. & Syst.

View full text Add to dashboard Cite

SUMMARYMapReduce is a parallel processing framework for large scale data. In the reduce phase, MapReduce employs the hash scheme in order to distribute data sharing the same key across cluster nodes. However, this approach is not robust for the skewed data distribution. In this paper, we propose a skew-tolerant key distribution method for MapReduce. The proposed method assigns keys to cluster nodes balancing their workloads. We implemented our proposed method on Hadoop. Through experiments, we evaluate the performance of the proposed method in comparison with the conventional method.

show abstract

“…Substantial research efforts have been dedicated to either adopting MapReduce in different environments such as multi-core [25], graphics processors (GPU)s [12], and virtual machines [15,30] or to improving MapReduce performance through skewhandling [16,21] and locality-execution [14,33].…”

Section: Related Workmentioning

confidence: 99%

Towards Efficient Power Management in MapReduce: Investigation of CPU-Frequencies Scaling on Power Efficiency in Hadoop

Ibrahim

Moise²,

Chihoub³

et al. 2014

Lecture Notes in Computer Science

Self Cite

View full text Add to dashboard Cite

Abstract. With increasingly inexpensive cloud storage and increasingly powerful cloud processing, the cloud has rapidly become the environment to store and analyze data. Most of the large-scale data computations in the cloud heavily rely on the MapReduce paradigm and its Hadoop implementation. Nevertheless, this exponential growth in popularity has significantly impacted power consumption in cloud infrastructures. In this paper, we focus on MapReduce and we investigate the impact of dynamically scaling the frequency of compute nodes on the performance and energy consumption of a Hadoop cluster. To this end, a series of experiments are conducted to explore the implications of Dynamic Voltage Frequency scaling (DVFS) settings on power consumption in Hadoopclusters. By adapting existing DVFS governors (i.e., performance, powersave, ondemand, conservative and userspace) in the Hadoop cluster, we observe significant variation in performance and power consumption of the cluster with different applications when applying these governors: the different DVFS settings are only sub-optimal for different MapReduce applications. Furthermore, our results reveal that the current CPU governors do not exactly reflect their design goal and may even become ineffective to manage the power consumption in Hadoop clusters. This study aims at providing more clear understanding of the interplay between performance and power management in Hadoop cluster and therefore offers useful insight into designing power-aware techniques for Hadoop systems.

show abstract

LEEN: Locality/Fairness-Aware Key Partitioning for MapReduce in the Cloud

Cited by 138 publications

References 11 publications

Improving MapReduce Performance in Heterogeneous Network Environments and Resource Utilization

Improving MapReduce Performance in Heterogeneous Network Environments and Resource Utilization

Skew-Tolerant Key Distribution for Load Balancing in MapReduce

Towards Efficient Power Management in MapReduce: Investigation of CPU-Frequencies Scaling on Power Efficiency in Hadoop

Contact Info

Product

Resources

About