Self-Balancing Job Parallelism and Throughput in Hadoop

Abstract

In Hadoop cluster, the performance and the resource consumption of MapReduce jobs do not only depend on the characteristics of these applications and workloads, but also on the appropriate setting of Hadoop configuration parameters. However, when the job workloads are not known a priori or they evolve over time, a static configuration may quickly lead to a waste of computing resources and consequently to a performance degradation. In this paper, we therefore propose an on-line approach that dynamically reconfigures Hadoop at runtime. Concretely, we focus on balancing the job parallelism and throughput by adjusting Hadoop capacity scheduler memory configuration. Our evaluation shows that the approach outperforms vanilla Hadoop deployments by up to 40% and the best statically profiled configurations by up to 13%.

Hadoop-Benchmark: Rapid Prototyping and Evaluation of Self-Adaptive Behaviors in Hadoop Clusters

“…This is one of the crucial parameter whose inappropriate configuration can have a serious impact on Hadoop performance (up to 40%, cf. Zhang et al [11]).…”

“…For this task, we rely on docker-machine 10 , a tool that creates virtual machines and install Docker engine on them. It currently supports creating virtual machines in local machine using either VirtualBox or VMWare, in local cluster, reusing existing machines simply via SSH, or in number of cloud providers 11 including all the major vendors. The main advantage of using Docker Machine is in the layer of abstraction it provides that can form the very same Hadoop cluster regardless the actual virtualization and cloud environment being used.…”

“…As a use case to show the capabilities of the platform, we choose a self-adaptive behavior that automatically balances the job parallelism and throughput in a Hadoop cluster [11]. Concretely, it adjusts a YARN capacity scheduler parameter, MARP 16 , which controls the ratio between the number of concurrently executing MapReduce jobs versus the number of running map and reduce tasks.…”

“…To demonstrate the usage of the platform, we include a complete implementation of a Hadoop self-adaptation case study. Concretely, a feedback control loop that balances Hadoop job parallelism and throughput through Hadoop capacity scheduler adjustment based on our previous work [11].…”

Turbocharging Hadoop Fair Scheduler using Dynamic Job Grouping in Multi-Job Workloads

“…They improved Hadoop's name node performance and proposed an optimization framework. Zhang et al [13] dynamically reconfigured Hadoop for realtime performance, outperforming the vanilla Hadoop. Lo and Cheng [14] proposed a modified fair scheduler that dynamically adjusts the resource allocation for user jobs and reduces the average turnaround time.…”