Towards Privacy Preserving LLMs Training

Abstract

Privacy-preserving training of machine learning models aims to avoid or minimize (mitigate) the exact or similar reproduction (leakage) of information contained in the training data. This chapter introduces pre-processing methods (filtering and de-duplication) that prepare the training data to minimize information leakage, followed by a discussion of training and deployment methods (differentially private fine-tuning, noisy knowledge transfer) that provide empirical or theoretical guarantees for the achieved privacy protection with a focus on Large Language Models (LLMs).