Deadline-Based Scheduling for GPU with Preemption Support

Abstract

Modern automotive-grade embedded computing platforms feature high-performance Graphics Processing Units (GPUs) to support the massively parallel processing power needed for next-generation autonomous driving applications (e.g., Deep Neural Network (DNN) inference, sensor fusion, path planning, etc). As these workload-intensive activities are pushed to higher criticality levels, there is a stronger need for more predictable scheduling algorithms that are able to guarantee predictability without overly sacrificing GPU utilization. Unfortunately, the real-rime literature on GPU scheduling mostly considered limited (or null) preemption capabilities, while previous efforts in broader domains were often based on programming models and APIs that were not designed to support the real-rime requirements of recurring workloads. In this paper, we present the design of a prototype real-time scheduler for GPU activities on an embedded System on a Chip (SoC) featuring a cuttingedge GPU architecture by NVIDIA adopted in the autonomous driving domain. The scheduler runs as a software partition on top of the NVIDIA hypervisor, and it leverages latest generation architectural features, such as pixel-level preemption and threadlevel preemption. Such a design allowed us to implement and test a preemptive Earliest Deadline First (EDF) scheduler for GPU tasks providing bandwidth isolations by means of a Constant Bandwidth Server (CBS). Our work involved investigating alternative programming models for compute APIs, allowing us to characterize CPU-to-GPU command submission with more detailed scheduling information. A detailed experimental characterization is presented to show the significant schedulability improvement of recurring real-time GPU tasks.

Timing isolation and improved scheduling of deep neural networks for real‐time systems

“…A case study based on a real implementation demonstrated improvements on both average-case (up to 24%) and longest-observed (up to 30%) response times of DNNs, with respect to the adoption of standard approaches. Possible future research directions target the design of predictable mechanisms for scheduling DNN on heterogeneous platforms, such as GPUs 29 and reprogrammable FPGAs, 53 and the extension of other protocols (eg, proxy execution 34,35 and BROE 38,39 ) to handle budget exhaustion inside critical sections to parallel workloads scheduled by reservation servers.…”

“…Similar mechanisms have been proposed by Capodieci et al 27 and Ali and Yun. 28 Capodieci et al 29 implemented the constant bandwidth server (also used in this paper to provide timing isolation of Tensorflow threads) for scheduling CUDA kernels on Nvidia GPUs. Unfortunately, their work is not publicly available.…”

Contending memory in heterogeneous SoCs: Evolution in NVIDIA Tegra embedded platforms

“…However, this is a parasite effect given by the TensorRT API implementation, which submits input and output network conversion layers to the iGPU. In this way, the measured latencies for the GPU are also perturbed by the GPU context scheduler [28].…”

Graphic Interfaces in ADAS

“…This allows the system designer to easily account for such periods and CPU/GPU processing time for scheduling purposes. We refer the reader to the following paper for an in-depth understanding of GPU scheduling on NVIDIA platforms [6].…”