DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Pass

Abstract

Differentially private stochastic gradient descent (DP-SGD) adds noise to gradients in back-propagation, safeguarding training data from privacy leakage, particularly membership inference. It fails to cover (inference-time) threats like embedding inversion and sensitive attribute inference. It is also costly in storage and computation when used to fine-tune large pre-trained language models (LMs).We propose DP-Forward, which directly perturbs embedding matrices in the forward pass of LMs. It satisfies stringent local DP requirements for training and inference data. To instantiate it using the smallest matrix-valued noise, we devise an analytic matrix Gaussian mechanism (aMGM) by drawing possibly non-i.i.d. noise from a matrix Gaussian distribution. We then investigate perturbing outputs from different hidden (sub-)layers of LMs with aMGM noises. Its utility on three typical tasks almost hits the non-private baseline and outperforms DP-SGD by up to 7.7pp at a moderate privacy level. It saves 3× time and memory costs compared to DP-SGD with the latest high-speed library. It also reduces the average success rates of embedding inversion and sensitive attribute inference by up to 88pp and 41pp, respectively, whereas DP-SGD fails. CCS CONCEPTS• Security and privacy → Data anonymization and sanitization; Privacy-preserving protocols; Privacy protections.

Differentially Private Fine-tuning of Language Models

“…Since the initial appearance of our paper, private fine-tuning (of language models and beyond) has become perhaps the standard paradigm for doing private machine learning in many settings (Bu et al, 2022;Wu et al, 2024;Du et al, 2023;Pelikan et al, 2023). He et al (2023) explored larger-scale settings, including private fine-tuning of GPT-3.…”