Workload characteristics and resource aware Hadoop scheduler

Abstract

Hadoop MapReduce is one of the largely used platforms for large scale data processing. Hadoop cluster has machines with different resources, including memory size, CPU capability and disk space. This introduces challenging research issue of improving Hadoop's performance through proper resource provisioning. The work presented in this paper focuses on optimizing job scheduling in Hadoop. Workload Characteristic and Resource Aware (WCRA) Hadoop scheduler is proposed, that classifies the jobs into CPU bound and Disk I/O bound. Based on the performance, nodes in the cluster are classified as CPU busy and Disk I/O busy. The amount of primary memory available in the node is ensured to be more than 25% before scheduling the job. Performance parameters of Map tasks such as the time required for parsing the data, map, sort and merge the result, and of Reduce task, such as the time to merge, parse and reduce is considered to categorize the job as CPU bound or Disk I/O bound. Tasks are assigned the priority based on their minimum Estimated Completion Time. The jobs are scheduled on a compute node in such a way that jobs already running on it will not be affected. Experimental results has given 30 % improvement in performance compared to Hadoop's FIFO, Fair and Capacity scheduler.

Hadoop MapReduce scheduling paradigms

“…3) Workload Characteristic and Resource Aware Scheduler: In this paper, the authors propose WCRA-scheduling of Hadoop clusters (Workload Characteristic and Resource Aware) [12]. WCRA-scheduling checks the CPU, RAM and I/O-load on the nodes first.…”

“…The work bears some similarity to [7], but also embraces RAM as an important parameter, ensuring that more than 25% of the primary memory is always available before scheduling a job. The authors argue that "is critical in case of CPU and Disk I/O bound tasks" [12]. It was found that "compute node works significantly if it has the available physical memory greater than 25%.…”

“…It was found that "compute node works significantly if it has the available physical memory greater than 25%. Tasks are assigned to the node if the memory availability is greater than 25%" [12]. WRCA-scheduling has the benefit of being specifically designed to handle heterogeneous clusters.…”

A Framework of Security and Performance Enhancement for Hadoop

“…So in AES-MR each map function is performed in such a way that it decrypts the entire encrypted file, chunk by chunk in the Map phase and forwards the decrypted result to the reducer phase to get the original file. [3].…”

“…The trend for larger data sets is due to the additional information which is derivable from analysis of one large set of related data, as compared to separate smaller sets with the same total amount of data. Big data is difficult to work with the most relational database management systems and desktop statistics and visualization packages, instead they require massively parallel softwares running on tens, hundreds, or thousands of servers [3].…”