Improving MapReduce Performance in Heterogeneous Network Environments and Resource Utilization

MapReduce has become a prominent Parallel processing model used for analysing large scale data. MapReduce applications are increasingly being deployed in the cloud along with other applications sharing the same physical resources. In this scenario, efficient scheduling of MapReduce applications is of utmost importance. Also, MapReduce has to consider various other parameters like energy efficiency and meeting SLA goals besides achieving performance when executing jobs in cloud environments. In this work, we have classified MapReduce Scheduling as Cluster based Scheduling and Objective based Scheduling. We then summarize and analyse the different class of schedulers highlighting the strong points and limitations of each of the scheduling approaches. The Adaptive scheduling techniques provide dynamic resource management and meet performance goals. The Energy efficient scheduling techniques aim to cut data centre costs by using different approaches. Finally, we discuss the current challenges and future work.

show abstract

“…scheduling [19] Resource stealing Proposes opportunistic use of the otherwise wasted resources. Does not consider user-specified goal.…”

Section: Heterogeneous Mapreduce Scheduling Techniquesmentioning

confidence: 99%

Analysis of MapReduce scheduling and its improvements in cloud environment

D’Souza

Chandrasekaran

2015

2015 IEEE International Conference on Signal Processing, Informatics, Communication and Energy Systems (SPICES)

View full text Add to dashboard Cite

show abstract

“…Z.Fadika et al [21] presented MARLA, a load-adaptive MapReduce framework espousing a task-oriented approach to MapReduce application processing faced with the source of cluster heterogeneity. [22] proposed resource stealing which enables running tasks to steal the unutilized resources and return them when new tasks were assigned. Although the above techniques can improve MapReduce performance of heterogeneous clusters, they do not take into account data locality and data movement overhead.…”

Section: Related Workmentioning

confidence: 99%

SLDP: A Novel Data Placement Strategy for Large-Scale Heterogeneous Hadoop Cluster

Luo¹,

Dong²

2014

2014 Second International Conference on Advanced Cloud and Big Data

View full text Add to dashboard Cite

Hadoop as a popular open-source implementation of MapReduce is widely used for large scale data-intensive applications like data mining, web indexing and scientific computing. The current Hadoop implementation assumes that nodes in a cluster are homogeneous in nature, and Hadoop distributed file system(HDFS) distributes data to multiple nodes based on disk space availability. Such data placement strategy is very efficient for homogeneous environments, where nodes are identical in terms of both computing power and disk capacity. Unfortunately, in practice, the homogeneity assumptions do not always hold. Hadoop's scheduler will lead to severe performance degradation and energy dissipation in heterogeneous environments by using default data placement strategy of HDFS. In this paper, we propose a novel snakelike data placement mechanism (SLDP) for large-scale heterogeneous Hadoop cluster. SLDP adopts a heterogeneity aware algorithm to divide various nodes into several virtual storage tiers(VST) firstly, and then places data blocks across nodes in each VST circuitously according to the hotness of data. Furthermore, SLDP uses a hotness proportional replication to reduce disk space consumption and also has an effective power control function. Experimental results on two real data-intensive applications show that SLDP is energy-efficient, space-saving and able to improve MapReduce performance in a heterogeneous Hadoop cluster significantly.

show abstract

“…Moreover, Jiang concludes that carefully tuning each factor, it is possible to eliminate the negative impact of these factors and improve the performance of MapReduce applications. Other authors like Guo [10] and Cheng [11] focus their works on improving the performance of MapReduce applications. Gou explodes the freedom to control concurrency in MapReduce in order to improve resource utilization.…”

Section: Related Workmentioning

confidence: 99%

Performance analysis model for big data applications in cloud computing

2014

View full text Add to dashboard Cite

The foundation of Cloud Computing is sharing computing resources dynamically allocated and released per demand with minimal management effort. Most of the time, computing resources such as processors, memory and storage are allocated through commodity hardware virtualization, which distinguish cloud computing from others technologies. One of the objectives of this technology is processing and storing very large amounts of data, which are also referred to as Big Data. Sometimes, anomalies and defects found in the Cloud platforms affect the performance of Big Data Applications resulting in degradation of the Cloud performance. One of the challenges in Big Data is how to analyze the performance of Big Data Applications in order to determine the main factors that affect the quality of them. The performance analysis results are very important because they help to detect the source of the degradation of the applications as well as Cloud. Furthermore, such results can be used in future resource planning stages, at the time of design of Service Level Agreements or simply to improve the applications. This paper proposes a performance analysis model for Big Data Applications, which integrates software quality concepts from ISO 25010. The main goal of this work is to fill the gap that exists between quantitative (numerical) representation of quality concepts of software engineering and the measurement of performance of Big Data Applications. For this, it is proposed the use of statistical methods to establish relationships between extracted performance measures from Big Data Applications, Cloud Computing platforms and the software engineering quality concepts.

show abstract

Improving MapReduce Performance in Heterogeneous Network Environments and Resource Utilization

Cited by 55 publications

References 16 publications

Analysis of MapReduce scheduling and its improvements in cloud environment

Analysis of MapReduce scheduling and its improvements in cloud environment

SLDP: A Novel Data Placement Strategy for Large-Scale Heterogeneous Hadoop Cluster

Performance analysis model for big data applications in cloud computing

Contact Info

Product

Resources

About