Enabling Rack-scale Confidential Computing using Heterogeneous Trusted Execution Environment

Abstract

With its huge real-world demands, large-scale confidential computing still cannot be supported by today's Trusted Execution Environment (TEE), due to the lack of scalable and effective protection of high-throughput accelerators like GPUs, FPGAs, and TPUs etc. Although attempts have been made recently to extend the CPU-like enclave to GPUs, these solutions require change to the CPU or GPU chips, may introduce new security risks due to the side-channel leaks in CPU-GPU communication and are still under the resource constraint of today's CPU TEE.To address these problems, we present the first Heterogeneous TEE design that can truly support large-scale compute or data intensive (CDI) computing, without any chip-level change. Our approach, called HETEE, is a device for centralized management of all computing units (e.g., GPUs and other accelerators) of a server rack. It is uniquely designed to work with today's data centres and clouds, leveraging modern resource pooling technologies to dynamically compartmentalize computing tasks, and enforce strong isolation and reduce TCB through hardware support. More specifically, HETEE utilizes the PCIe ExpressFabric to allocate its accelerators to the server node on the same rack for a non-sensitive CDI task, and move them back into a secure enclave in response to the demand for confidential computing. Our design runs a thin TCB stack for security management on a security controller (SC), while leaving a large set of software (e.g., AI runtime, GPU driver, etc.) to the integrated microservers that operate enclaves. An enclaves is physically isolated from others through hardware and verified by the SC at its inception. Its microserver and computing units are restored to a secure state upon termination.We implemented HETEE on a real hardware system, and evaluated it with popular neural network inference and training tasks. Our evaluations show that HETEE can easily support the CDI tasks on the real-world scale and incurred a maximal throughput overhead of 2.17% for inference and 0.95% for training on ResNet152.Recent years have seen attempts to support the heterogeneous TEE. Examples include Graviton [15] and HIX [16]. However, all these approaches require changes to CPU and (or) GPU chips, which prevents the use of existing hardware, and also incurs a long and expensive development cycle to chip manufacturers and therefore may not happen in the near 1450 2020 IEEE Symposium on Security and Privacy

Case Study: Spark GPU-Enabled Framework to Control COVID-19 Spread Using Cell-Phone Spatio-Temporal Data

“…The effect of using a GPU node in this case study is noticable. GPU is more oriented to compute-intensive tasks rather than data-intensive tasks because the cost of data transfer between GPU memory and the host memory is expensive and time-consuming [Zhu, Hou, Wang et al (2019)]. Hence, data cleansing and preprocessing are executed by CPU nodes only.…”

ShEF: shielded enclaves for cloud FPGAs

“…HIX [54] separated the driver out from the OS and ran it inside a trusted CPU enclave, essentially creating a heterogeneous trusted environment across both CPU and GPU, but requiring changes to the CPU and PCIe root complex. HETEE [105] proposed fabricating a tamper-resistant box of accelerators (namely GPUs) that a rack of servers can access via a centralized security controller in a dedicated FPGA. However, it did not consider the unique challenges that arise in cloud FPGAs (Section 2.4) and required a specialized tamper-resistant chassis.…”

“…Unfortunately, recently-proposed TEEs for accelerators, including FPGAs, are either insecure against direct physical attacks [54,105], require fundamental hardware changes [54,94,105], only address isolated challenges such as attestation [36,52,72,99], or rely on external CPU TEEs [52,54,94]. Moreover, they ignore the Shell logic [3,57,60,88], a fundamental untrusted operating system for cloud FPGA logic.…”

Security and Trust in the 6G Era

“…It has been shown that HW assisted TEEs can improve security in a distributed cloud environment and with low performance overhead [28]. The concept of open framework and elastic scaling of TEEs on edge platforms needs to be analyzed more; first studies indicate that privacy and trust can be provided by scalable TEEs for heterogeneous systems (such as a combination of CPUs and GPUs) for performing data intensive computation [29] as needed for demanding 6G use cases such as mixed and augmented reality. Building on the principle of transitive trust [27], TEEs can provide attestation of trust anchored in the confirmed genuineness of enclaves on different processing unit (xPU) levels and of direct relevance to application and user.…”