AvA: Accelerated Virtualization of Accelerators

Abstract

Applications are migrating en masse to the cloud, while accelerators such as GPUs, TPUs, and FPGAs proliferate in the wake of Moore's Law. These trends are in conflict: cloud applications run on virtual platforms, but existing virtualization techniques have not provided production-ready solutions for accelerators. As a result, cloud providers expose accelerators by dedicating physical devices to individual guests. Multi-tenancy and consolidation are lost as a consequence.We present AvA, which addresses limitations of existing virtualization techniques with automated construction of hypervisor-managed virtual accelerator stacks. AvA combines a DSL for describing APIs and sharing policies, deviceagnostic runtime components, and a compiler to generate accelerator-specific components such as guest libraries and API servers. AvA uses Hypervisor Interposed Remote Acceleration (HIRA), a new technique to enable hypervisorenforcement of sharing policies from the specification.We use AvA to virtualize nine accelerators and eleven framework APIs, including six for which no virtualization support has been previously explored. AvA provides nearnative performance and can enforce sharing policies that are not possible with current techniques, with orders of magnitude less developer effort than required for hand-built virtualization support.CCS Concepts • Software and its engineering Virtual machines; Operating systems; Source code generation.

GPUReplay: a 50-KB GPU stack for client ML

“…to a hypervisor [95] or to a remote server [34]. The interposed interfaces include GPU APIs [34,108] and GPU MMIO [33,95]. Notably, AvA [108] records and replays API calls during GPU VM migration.…”

“…The interposed interfaces include GPU APIs [34,108] and GPU MMIO [33,95]. Notably, AvA [108] records and replays API calls during GPU VM migration. GR shares the principle of interposition and gives it a new use ś for recording computations ahead of time and later replaying it on a different machine.…”

Bertha: Tunneling through the Network API

“…Our work is enabled by the long line of prior work on active networking [60], network function virtualization (NFV) [16], and programmable switches and SmartNICs [10,31,37,53,58] that enable programmability in the network fast path. Recent work on virtualizing programmable network devices and accelerators, including HyPer4 [25], P4Visor [65], AmorphOS [32] and AvA [63] have made it easier to share these offloads among applications. Our focus in this work has been on simplifying the use of these advances in applications.…”

Debugging in the brave new world of reconfigurable hardware

“…Specialized hardware designs provide unprecedented efficiency in domains such as machine learning [74,83,101,102,122,127,128], compression [92,125], database operations [88,96,104], graph processing [36,47,112,129], networking [41,52,111], and storage virtualization [78]. To realize the benefits of FPGAs, systems researchers have built operating systems [53,73,77,106], virtualization support [42,46,80,85,113,120,123,124], just-in-time compilers [97], and high-level synthesis tools [43,44,61,116,117]. The proliferation and benefits of FPGAs have even prompted major cloud vendors to provide FPGA instances on their platforms [31,33].…”