On the usability of shortest remaining time first policy in shared Hadoop clusters

Abstract

Hadoop has been recently used to process a diverse variety of applications, sharing the same execution infrastructure. A practical problem facing the Hadoop community is how to reduce job makespans by reducing job waiting times and execution times. Previous Hadoop schedulers have focused on improving job execution times, by improving data locality but not considering job waiting times. Even worse, enforcing data locality according to the job input sizes can be inefficient: it can lead to long waiting times for small yet short jobs when sharing the cluster with jobs with smaller input sizes but higher execution complexity. This paper presents hSRTF, an adaption of the well-known Shortest Remaining Time First scheduler (i.e., SRTF) in shared Hadoop clusters. hSRTF embraces a simple model to estimate the remaining time of a job and a preemption primitive (i.e., kill) to free the resources when needed. We have implemented hSRTF and performed extensive evaluations with Hadoop on the Grid'5000 testbed. The results show that hSRTF can significantly reduce the waiting times of small jobs and therefore improves their makespans, but at the cost of a relatively small increase in the makespans of large jobs. For instance, a time-based proportional share mode of hSRTF (i.e., hSRTF-Pr) speeds up small jobs by (on average) 45% and 26% while introducing a performance degradation for large jobs by (on average) 10% and 0.2% compared to Fifo and Fair schedulers, respectively. CCS Concepts •Computer systems organization → Distributed architectures; •Networks → Cloud computing;

Simulator for Scheduling Real-Time Systems with Reduced Power Consumption

“…Shortest Remaining Time (SRT) scheduling focuses on the burst time of the task, but it the work of the scheduling is different from LRT. This type of scheduling focuses on the remaining time of the burst time, and it arranges the priority of the tasks according to the least remaining time [24]. So with that said, the process with minimum left over time is considered as prior than the one with more left over time.…”

POSUM: A Portfolio Scheduler for MapReduce Workloads

“…However, delay scheduling approaches [25] [3], relax inter-job priority even more to achieve better data locality. Cheriere et al [26] argue that considering data locality when choosing which job should run on a given free slot leads to long wait times for small short jobs. Their Shortest Remaining Time First (SRTF) scheduling policies give higher priority to such jobs.…”

“…The resulting set of four policies thus achieves a balance Share-based version of the Shortest-Remaining-Time-First scheduler [26].…”