SAF: Simulated Annealing Fair Scheduling for Hadoop Yarn Clusters

Abstract

Apache introduced YARN as the next generation of the Hadoop framework, providing resource management and a central platform to deliver consistent data governance tools across Hadoop clusters. Hadoop YARN supports multiple frameworks like MapReduce to process different types of data and works with different scheduling policies such as FIFO, Capacity, and Fair schedulers. DRF is the best option that uses short-term, without considering history information, convergence to fairness for multi-type resource allocation. However, DRF performance is still not satisfying due to trade-offs between fairness and performance regarding resource utilization.To address this problem, we propose Simulated Annealing Fair scheduling, SAF, a long-term fair scheme in resource allocation to have fairness and excellent performance in terms of resource utilization and MakeSpan. We introduce a new parameter as entropy, which is an approach to indicates the disorder in the fairness of allocated resources of the whole cluster. We implemented SAF as a pluggable scheduler in Hadoop Yarn Cluster and evaluated it with standard MapReduce benchmarks in Yarn Scheduler Load Simulator (SLS) and CloudSim Plus simulation framework. Finally, the results of both simulation tools are evidence to prove our claim. Compared to DRF, SAF increases resource utilization of YARN clusters significantly and decreases MakeSpan to an appropriate level.

New efficient Hadoop scheduler: Generalized particle swarm optimization and simulated annealing‐dominant resource fairness

“…The default policies in the YARN scheduler has a dominant resource fairness policy (DRF) which is FAIR policy with the high weightage given to the most resource-demanding node. 7 It keeps an eye on the minimum required resources by each instance. If any instance has already allotted the minimum required share (minshare) of resources, then the request is dropped irrespective of priority.…”