Enabling preemptive multiprogramming on GPUs

Abstract

GPUs are being increasingly adopted as compute accelerators in many domains, spanning environments from mobile systems to cloud computing. These systems are usually running multiple applications, from one or several users. However GPUs do not provide the support for resource sharing traditionally expected in these scenarios. Thus, such systems are unable to provide key multiprogrammed workload requirements, such as responsiveness, fairness or quality of service.In this paper, we propose a set of hardware extensions that allow GPUs to efficiently support multiprogrammed GPU workloads. We argue for preemptive multitasking and design two preemption mechanisms that can be used to implement GPU scheduling policies. We extend the architecture to allow concurrent execution of GPU kernels from different user processes and implement a scheduling policy that dynamically distributes the GPU cores among concurrently running kernels, according to their priorities. We extend the NVIDIA GK110 (Kepler) like GPU architecture with our proposals and evaluate them on a set of multiprogrammed workloads with up to eight concurrent processes. Our proposals improve execution time of high-priority processes by 15.6x, the average application turnaround time between 1.5x to 2x, and system fairness up to 3.4x.

Chimera

“…Moreover, each SM has its own on-chip scratch-pad memory, which is shared by the threads within a thread block. For modern GPUs, the context of a single SM can be as large as 256kB of register file and 48kB of shared memory [1,24,29]. With such a large context, preempting with context switching has high overhead in both preemption latency and wasted throughput.…”

“…However, supporting preemptive multitasking on GPUs through context switching can incur a higher overhead compared to CPUs, where the context of an SM can be as large as 256kB of register file and 48kB of on-chip scratch-pad memory [1,24,29]. Not only does a kernel have to endure a long preemption latency, the GPU also wastes execution resources while context switching.…”

“…Not only does a kernel have to endure a long preemption latency, the GPU also wastes execution resources while context switching. Although Tanasic et al [29] has shown that the average normalized turnaround time can still be improved with high context switching overhead, such overhead wastes the GPU's computing power and may be ineffective for latency-sensitive applications [3,12,13].…”

“…Context switching [17,29] stores the context of currently running thread blocks, and preempts an SM with a new kernel. Draining [12,29] stops issuing new thread blocks to the SM and waits until the SM finishes its currently running thread blocks. Flushing drops the execution of running thread blocks and preempts the SM almost instantly.…”

Efficient GPU multitasking with latency minimization and cache boosting

“…As many applications do not require full GPU resources, spatial multitasking can improve total system throughput with concurrent execution of multiple applications compared to temporal multitasking [12,14].…”