Over 100x Faster Bootstrapping in Fully Homomorphic Encryption through Memory-centric Optimization with GPUs

Abstract

Fully Homomorphic encryption (FHE) has been gaining in popularity as an emerging means of enabling an unlimited number of operations in an encrypted message without decryption. A major drawback of FHE is its high computational cost. Specifically, a bootstrapping step that refreshes the noise accumulated through consequent FHE operations on the ciphertext can even take minutes of time. This significantly limits the practical use of FHE in numerous real applications.By exploiting the massive parallelism available in FHE, we demonstrate the first instance of the implementation of a GPU for bootstrapping CKKS, one of the most promising FHE schemes supporting the arithmetic of approximate numbers. Through analyzing CKKS operations, we discover that the major performance bottleneck is their high main-memory bandwidth requirement, which is exacerbated by leveraging existing optimizations targeted to reduce the required computation. These observations motivate us to utilize memory-centric optimizations such as kernel fusion and reordering primary functions extensively.Our GPU implementation shows a 7.02× speedup for a single CKKS multiplication compared to the state-of-the-art GPU implementation and an amortized bootstrapping time of 0.423us per bit, which corresponds to a speedup of 257× over a single-threaded CPU implementation. By applying this to logistic regression model training, we achieved a 40.0× speedup compared to the previous 8-thread CPU implementation with the same data.

Does Fully Homomorphic Encryption Need Compute Acceleration?

“…In Table VII, we give benchmarks for the logistic regression implementation based on our architecture modeling discussed in Section II-E. The parameters we use are from the work of Jung et al [22], and these parameters were chosen to optimize their secure logistic regression application that leverages a GPU implementation of CKKS bootstrapping. We refer to the original work of Han et al [19] for the full algorithm benchmarked in Table VII.…”

“…Key Takeaway: Bootstrapping is often the bottle-neck operation in HE applications, especially applications that implement a deep circuit. For example, even when using a heavilyoptimized GPU implementation of bootstrapping, nearly half of the time in HE logistic regression training is spent on bootstrapping [22] (table VII). This motivates the need to optimize the Bootstrap operation to efficiently support deep circuits.…”

“…Real-world applications commonly attempt to amortize this bootstrapping cost across multiple homomorphic operations. Even when considering these application-specific optimizations, bootstrapping consumes more than 50% of the total compute and memory budget for end-to-end operations like machine learning training [22]. To make FHE-based comput-ing practical, we need to consider a multi-layer approach to accelerate both the bootstrapping step as well as its primitive building blocks using a combination of algorithmic and hardware techniques.…”

“…These include both linear and non-linear operation optimizations proposed by Han and Ki [20], Han, Hhan and Cheon [18], and Bossuat, Mouchet, Troncoso-Pastoriza and Hubaux [3]. Recent bootstrapping implementation on GPUs by Jung, Kim, Ahn, Cheon and Lee [22] is the first work to perform memory-centric optimizations for linear operations in bootstrapping. Even after these optimizations, their implementation continues to be bounded by main memory bandwidth and exhibits an arithmetic intensity of < 1 Op/byte.…”

Precise Approximation of Convolutional Neural Networks for Homomorphically Encrypted Data

“…The performance of FHE techniques has been improved a lot enough to be used in practical applications. For example, Cheon-Kim-Kim-Song (CKKS) scheme [12], one of the representative word-wise FHE schemes, has been improved much in recent years by bootstrapping algorithm improvements [13,14,15] and GPU implementations [16]. Furthermore, significant improvements in bootstrapping accuracy have resulted in achieving a precision of up to 40 bits [13], and laid a good foundation for performing deep learning on FHE for practical uses.…”