Truth Serum

Tramèr, Florian; Shokri, Reza; Joaquin, Ayrton San; Le, Hoang Hai; Jagielski, Matthew; Hong, Sanghyun; Carlini, Nicholas

doi:10.1145/3548606.3560554

Cited by 32 publications

(1 citation statement)

References 32 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…some training examples are intrinsically easier/harder to learn than other examples and thus have higher/lower accuracy). Existing works train "reference" models [4,5] to calibrate the hardness of different training examples to rule out the variance and bring out the subtle difference between memorization and generalization. However, these works have to train tens to hundreds of reference models for obtaining good calibration.…”

Section: Introductionmentioning

confidence: 99%

Detecting Unintended Memorization in Language-Model-Fused ASR

Huang¹,

Chien²,

Thakkar³

et al. 2022

Interspeech 2022

View full text Add to dashboard Cite

It is well-known that neural networks can unintentionally memorize their training examples, causing privacy concerns. However, auditing memorization in large non-auto-regressive automatic speech recognition (ASR) models has been challenging due to the high compute cost of existing methods such as hardness calibration. In this work, we design a simple auditing method to measure memorization in large ASR models without the extra compute overhead. Concretely, we speed up randomly-generated utterances to create a mapping between vocal and text information that is difficult to learn from typical training examples. Hence, accurate predictions only for sped-up training examples can serve as clear evidence for memorization, and the corresponding accuracy can be used to measure memorization. Using the proposed method, we showcase memorization in the state-of-the-art ASR models. To mitigate memorization, we tried gradient clipping during training to bound the influence of any individual example on the final model. We empirically show that clipping each example's gradient can mitigate memorization for sped-up training examples with up to 16 repetitions in the training set. Furthermore, we show that in large-scale distributed training, clipping the average gradient on each compute core maintains neutral model quality and compute cost while providing strong privacy protection.

show abstract

Section: Introductionmentioning

confidence: 99%

Detecting Unintended Memorization in Language-Model-Fused ASR

Huang¹,

Chien²,

Thakkar³

et al. 2022

Interspeech 2022

View full text Add to dashboard Cite

show abstract

Membership Inference Attack Against Principal Component Analysis

Zari

Parra-Arnau

Unsal

et al. 2022

Privacy in Statistical Databases

View full text Add to dashboard Cite

This paper studies the performance of membership inference attacks against principal component analysis (PCA). In this attack, we assume that the adversary has access to the principal components, and her main goal is to infer whether a given data sample was used to compute these principal components. We show that our attack is successful and achieves high performance when the number of samples used to compute the principal components is small. As a defense strategy, we investigate the use of various differentially private mechanisms. Accordingly, we present experimental results on the performance of Gaussian and Laplace mechanisms under naive and advanced compositions against MIA as well as the utility of these differentially-private PCA solutions.

show abstract

Towards Privacy Preserving LLMs Training

Buesser

2024

Large Language Models in Cybersecurity

View full text Add to dashboard Cite

Privacy-preserving training of machine learning models aims to avoid or minimize (mitigate) the exact or similar reproduction (leakage) of information contained in the training data. This chapter introduces pre-processing methods (filtering and de-duplication) that prepare the training data to minimize information leakage, followed by a discussion of training and deployment methods (differentially private fine-tuning, noisy knowledge transfer) that provide empirical or theoretical guarantees for the achieved privacy protection with a focus on Large Language Models (LLMs).

show abstract

Truth Serum

Cited by 32 publications

References 32 publications

Detecting Unintended Memorization in Language-Model-Fused ASR

Detecting Unintended Memorization in Language-Model-Fused ASR

Membership Inference Attack Against Principal Component Analysis

Towards Privacy Preserving LLMs Training

Contact Info

Product

Resources

About