Peacock: Probe-Based Scheduling of Jobs by Rotating Between Elastic Queues

Abstract

In this paper, we propose Peacock, a new distributed probebased scheduler which handles heterogeneous workloads in data analytics frameworks with low latency. Peacock mitigates the Head-of-Line blocking problem, i.e., shorter tasks are enqueued behind the longer tasks, better than the state-of-the-art. To this end, we introduce a novel probe rotation technique. Workers form a ring overlay network and rotate probes using elastic queues. It is augmented by a novel probe reordering algorithm executed in workers. We evaluate the performance of Peacock against two state-of-the-art probe-based solutions through both tracedriven simulation and distributed experiment in Spark under various loads and cluster sizes. Our large-scale performance results indicate that Peacock outperforms the state-of-the-art in all cluster sizes and loads. Our distributed experiments confirm our simulation results.

TEBAS: A Time-Efficient Balance-Aware Scheduling Strategy for Batch Processing Jobs

“…Considering computing tasks of the BP job commonly demand the same resource type, these scheduling strategies can easily suffer from resource contention caused by the task co-location. To address these problems, several distributed schedulers are proposed to reduce scheduling overhead while improving load balancing performance [10], [11]. However, this divide-and-conquer approach fails to maintain a global view of cluster resources, resulting in a sub-optimal assignment.…”