A Micro-benchmark Suite for Evaluating Hadoop MapReduce on High-Performance Networks

Shankar, Dipti; Lu, Xiaoyi; Wasi-ur-Rahman, Md.; Islam, Nusrat Sharmin; Panda, Dhabaleswar K.

doi:10.1007/978-3-319-13021-7_2

Cited by 7 publications

(2 citation statements)

References 15 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…3. In addition to the above benchmarks that address the Hadoop framework as a whole, several microbenchmark suites have been designed to study individual components of the Apache Hadoop framework, such as Hadoop RPC [34], and Hadoop Distributed File Systems (HDFS) [24], and particularly Hadoop MapReduce [49], an extended version of which is presented in this paper.…”

Section: Background and Related Workmentioning

confidence: 99%

“…As an extension of the stand-alone MapReduce micro-benchmarks introduced by us in [49], which includes preliminary designs and results, this paper presents the motivation behind these micro-benchmarks, challenges in designing these microbenchmarks, and the workload characterization studies that were the basis for the various design considerations, based on popular comprehensive benchmark suites [20,56]. We present more detailed results, of experiments with the latest High-Performance RDMA-enhanced Design of YARN MapReduce on Modern HPC Clusters (HOMR) [44] over share-nothing, and shared storage architectures.…”

Section: Contributionsmentioning

confidence: 99%

See 1 more Smart Citation

Characterizing and benchmarking stand-alone Hadoop MapReduce on modern HPC clusters

Shankar

Wasi-ur-Rahman

et al. 2016

J Supercomput

Self Cite

View full text Add to dashboard Cite

With the emergence of high-performance data analytics, the Hadoop platform is being increasingly used to process data stored on high-performance computing clusters. While there is immense scope for improving the performance of Hadoop MapReduce (including the network-intensive shuffle phase) over these modern clusters, that are equipped with high-speed interconnects such as InfiniBand and 10/40 GigE, and storage systems such as SSDs and Lustre, it is essential to study the MapReduce component in an isolated manner. In this paper, we study popular MapReduce workloads, obtained from well-accepted, comprehensive benchmark suites, to identify common shuffle data distribution patterns. We determine different environmental and workload-specific factors that affect the performance of the MapReduce job. Based on these characterization studies, we propose a microbenchmark suite that can be used to evaluate the performance of stand-alone Hadoop MapReduce, and demonstrate its ease-of-use with different networks/protocols, Hadoop distributions, and storage architectures. Performance evaluations with our proposed micro-benchmarks show that stand-alone Hadoop MapReduce over IPoIB performs better than 10 GigE by about 13-15 %, and the RDMA-enhanced hybrid MapReduce design can achieve up to 43 % performance improvement over default Hadoop MapReduce over IPoIB, in both shared-nothing and shared storage architectures.

show abstract

Section: Background and Related Workmentioning

confidence: 99%

Section: Contributionsmentioning

confidence: 99%

Characterizing and benchmarking stand-alone Hadoop MapReduce on modern HPC clusters

Shankar

Wasi-ur-Rahman

et al. 2016

J Supercomput

Self Cite

View full text Add to dashboard Cite

show abstract

BigDataBench-MT: A Benchmark Tool for Generating Realistic Mixed Data Center Workloads

Han

Zhan²,

Shao

et al. 2016

Big Data Benchmarks, Performance Optimization, and Emerging Hardware

View full text Add to dashboard Cite

Abstract. Long-running service workloads (e.g. web search engine) and short-term data analysis workloads (e.g. Hadoop MapReduce jobs) colocate in today's data centers. Developing realistic benchmarks to reflect such practical scenario of mixed workload is a key problem to produce trustworthy results when evaluating and comparing data center systems. This requires using actual workloads as well as guaranteeing their submissions to follow patterns hidden in real-world traces. However, existing benchmarks either generate actual workloads based on probability models, or replay real-world workload traces using basic I/O operations. To fill this gap, we propose a benchmark tool that is a first step towards generating a mix of actual service and data analysis workloads on the basis of real workload traces. Our tool includes a combiner that enables the replaying of actual workloads according to the workload traces, and a multi-tenant generator that flexibly scales the workloads up and down according to users' requirements. Based on this, our demo illustrates the workload customization and generation process using a visual interface. The proposed tool, called BigDataBench-MT, is a multi-tenant version of our comprehensive benchmark suite BigDataBench and it is publicly available from http://prof.ict.ac.cn/ BigDataBench/multi-tenancyversion/.

show abstract

Accelerating Big Data Processing on Modern HPC Clusters

Lu¹,

Wasi-ur-Rahman²,

Islam³

et al. 2016

Conquering Big Data With High Performance Computing

View full text Add to dashboard Cite

A Micro-benchmark Suite for Evaluating Hadoop MapReduce on High-Performance Networks

Cited by 7 publications

References 15 publications

Characterizing and benchmarking stand-alone Hadoop MapReduce on modern HPC clusters

Characterizing and benchmarking stand-alone Hadoop MapReduce on modern HPC clusters

BigDataBench-MT: A Benchmark Tool for Generating Realistic Mixed Data Center Workloads

Accelerating Big Data Processing on Modern HPC Clusters

Contact Info

Product

Resources

About