Improving Performance of Heterogeneous MapReduce Clusters with Adaptive Task Tuning

Cheng, Dazhao; Rao, Jia; Guo, Yanfei; Jiang, Changjun; Zhou, Xiaobo

doi:10.1109/tpds.2016.2594765

Cited by 81 publications

(30 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Authors claim that the makespan is improved over 20 % compared to the classical MapReduce schedulers. In a virtual environment with heterogeneous capacities, containers for heterogeneous jobs are dynamically decided at runtime by Cheng et al and improved latency and resource utilization by 20 % , and 1 % , respectively. Two classes of algorithms are proposed in the work of Tang et al to minimize makespan and total completion time for a batch of workloads in a virtual environment.…”

Section: Literature Surveymentioning

confidence: 99%

“…In summary, works by Hu et al and Bei et al for heterogeneous MapReduce workloads in heterogeneous VM capacities try to minimize the makespan and improve resource utilization. More specifically, other works tune configuration parameters to exploit the heterogeneous capacity available in the virtual nodes, whereas Hu et al and Ming‐Chang et al classify the jobs to schedule map/reduce tasks based on the computing performance of a node. The significant shortcomings of these works are as follows: Tuning configuration parameters dynamically is job specific. Data locality is affected while concentrating on the performance of a node to place tasks. …”

Section: Literature Surveymentioning

confidence: 99%

See 1 more Smart Citation

Improving MapReduce scheduler for heterogeneous workloads in a heterogeneous environment

Jeyaraj

Ananthanarayana

Paul

2019

Concurrency and Computation

View full text Add to dashboard Cite

Summary Big data is largely influencing business entities and research sectors to be more data‐driven. Hadoop MapReduce is one of the cost‐effective ways to process large scale datasets and offered as a service over the Internet. Even though cloud service providers promise an infinite amount of resources available on‐demand, it is inevitable that some of the hired virtual resources for MapReduce are left unutilized and makespan is limited due to various heterogeneities that exist while offering MapReduce as a service. As MapReduce v2 allows users to define the size of containers for the map and reduce tasks, jobs in a batch become heterogeneous and behave differently. Also, the different capacity of virtual machines in the MapReduce virtual cluster accommodate a varying number of map/reduce tasks. These factors highly affect resource utilization in the virtual cluster and the makespan for a batch of MapReduce jobs. Default MapReduce job schedulers do not consider these heterogeneities that exist in a cloud environment. Moreover, virtual machines in MapReduce virtual cluster process an equal number of blocks regardless of their capacity, which affects the makespan. Therefore, we devised a heuristic‐based MapReduce job scheduler that exploits virtual machine and MapReduce workload level heterogeneities to improve resource utilization and makespan. We proposed two methods to achieve this: (i) roulette wheel scheme based data block placement in heterogeneous virtual machines, and (ii) a constrained 2‐dimensional bin packing to place heterogeneous map/reduce tasks. We compared heuristic‐based MapReduce job scheduler against the classical fair scheduler in MapReduce v2. Experimental results showed that our proposed scheduler improved makespan and resource utilization by 45.6% and 47.9% over classical fair scheduler.

show abstract

Section: Literature Surveymentioning

confidence: 99%

Section: Literature Surveymentioning

confidence: 99%

Improving MapReduce scheduler for heterogeneous workloads in a heterogeneous environment

Jeyaraj

Ananthanarayana

Paul

2019

Concurrency and Computation

View full text Add to dashboard Cite

show abstract

“…This method achieves this by dividing the nodes into a number of homogeneous sub-clusters and applies a self-tuning algorithm on individual sub-clusters. The experimental results by Dazhao Cheng et al (2017) show that their flex slot technique reduces the job completion time by 46% compared to the stock Hadoop cluster and by 22% compared to Skew Tune. Shanjiang Tang et al (2016) proposed a new fair resource allocation mechanism, the Long-Term Resource Fairness (LTRF).…”

Section: Review Of Literaturementioning

confidence: 99%

Design of Comprehensive Framework on Optimization Methods in Distributed Clusters

Pulamolu¹,

Subramanian²,

Krishnaraj³

2018

EAI Endorsed Transactions on Energy Web

View full text Add to dashboard Cite

“…In the paper [14], authors present new centroids initialization approach to improving the basic k-means algorithm with high-quality clusters. Authors in papers [22][23][24][25][26][27][28][29][30][31][32][33][34][35][36][37][38][39][40] have tried to improve the clustering algorithms which are used in various domains like networking and biometrics. However, these algorithms can be improved further.…”

Section: Related Workmentioning

confidence: 99%

Performance Analysis of Improved Clustering Algorithm on Real and Synthetic Data

Khandare¹,

Alvi²

2017

IJCNIS

View full text Add to dashboard Cite

Abstract-Clustering is an important technique in data mining to partition the data objects into clusters. It is a way to generate groups from the data objects. Different data clustering methods or algorithms are discussed in the various literature. Some of these are efficient while some are inefficient for large data. The k-means, Partition Around Method (PAM) or k-medoids, hierarchical and DBSCAN are various clustering algorithms. The k-means algorithm is more popular than the other algorithms used to partition data into k clusters. For this algorithm, k should be provided explicitly. Also, initial means are taken randomly but this may generate clusters with poor quality. This paper is a study and implementation of an improved clustering algorithm which automatically predicts the value of k and uses a new technique to take initial means. The performance analysis of the improved algorithm and other algorithms by using real and dummy datasets is presented in this paper. To measure the performance of algorithms, this paper uses running time of algorithms and various cluster validity measures. Cluster validity measures include sum squared error, silhouette score, compactness, separation, Dunn index and DB index. Also, the k predicted by the improved algorithm is compared with optimal k suggested by elbow method. It is found that both values of k are almost similar. Most of the values of validity measures for the improved algorithm are found to be optimal.

show abstract

Improving Performance of Heterogeneous MapReduce Clusters with Adaptive Task Tuning

Cited by 81 publications

References 14 publications

Improving MapReduce scheduler for heterogeneous workloads in a heterogeneous environment

Improving MapReduce scheduler for heterogeneous workloads in a heterogeneous environment

Design of Comprehensive Framework on Optimization Methods in Distributed Clusters

Performance Analysis of Improved Clustering Algorithm on Real and Synthetic Data

Contact Info

Product

Resources

About