User-Level Label Leakage from Gradients in Federated Learning

Wainakh, Aidmar; Ventola, Fabrizio; Müßig, Till; Keim, Jens; Cordero, Carlos García; Zimmer, Ephraim; Grube, Tim; Kersting, Kristian; Mühlhäuser, Max

doi:10.48550/arxiv.2105.09369

Cited by 10 publications

(15 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In this scenario, the user does not have to send updated batch norm statistics to the server. We further assume, for simplicity, that label information is contained in the update as metadata, but note that novel label recovery algorithms as discussed in Wainakh et al (2021) are highly successful, even at large aggregation sizes.…”

Section: B Technical Detailsmentioning

confidence: 99%

Fishing for User Data in Large-Batch Federated Learning via Gradient Magnification

Wen¹,

Geiping²,

Fowl³

et al. 2022

Preprint

View full text Add to dashboard Cite

Federated learning (FL) has rapidly risen in popularity due to its promise of privacy and efficiency. Previous works have exposed privacy vulnerabilities in the FL pipeline by recovering user data from gradient updates. However, existing attacks fail to address realistic settings because they either 1) require a 'toy' settings with very small batch sizes, or 2) require unrealistic and conspicuous architecture modifications. We introduce a new strategy that dramatically elevates existing attacks to operate on batches of arbitrarily large size, and without architectural modifications. Our model-agnostic strategy only requires modifications to the model parameters sent to the user, which is a realistic threat model in many scenarios. We demonstrate the strategy in challenging large-scale settings, obtaining high-fidelity data extraction in both cross-device and cross-silo federated learning. * Authors contributed equally. Order chosen randomly.

show abstract

Section: B Technical Detailsmentioning

confidence: 99%

Fishing for User Data in Large-Batch Federated Learning via Gradient Magnification

Wen¹,

Geiping²,

Fowl³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Many works have investigated what an attacker can infer from inspecting the intermediate gradients in FL settings or from inspecting multiple model snapshots during training [22,23,24,25,26]. These attacks focus on inferring training points, their labels, or related properties.…”

Section: E Further Related Workmentioning

confidence: 99%

Reconstructing Training Data with Informed Adversaries

Balle¹,

Cherubin²,

Hayes³

2022

Preprint

View full text Add to dashboard Cite

Given access to a machine learning model, can an adversary reconstruct the model's training data? This work studies this question from the lens of a powerful informed adversary who knows all the training data points except one. By instantiating concrete attacks, we show it is feasible to reconstruct the remaining data point in this stringent threat model. For convex models (e.g. logistic regression), reconstruction attacks are simple and can be derived in closed-form. For more general models (e.g. neural networks), we propose an attack strategy based on training a reconstructor network that receives as input the weights of the model under attack and produces as output the target data point. We demonstrate the effectiveness of our attack on image classifiers trained on MNIST and CIFAR-10, and systematically investigate which factors of standard machine learning pipelines affect reconstruction success. Finally, we theoretically investigate what amount of differential privacy suffices to mitigate reconstruction attacks by informed adversaries. Our work provides an effective reconstruction attack that model developers can use to assess memorization of individual points in general settings beyond those considered in previous works (e.g. generative language models or access to training gradients); it shows that standard models have the capacity to store enough information to enable high-fidelity reconstruction of training data points; and it demonstrates that differential privacy can successfully mitigate such attacks in a parameter regime where utility degradation is minimal.

show abstract

“…Depending on the model architecture, this frequency estimation can either be triggered by analyzing the bias gradient of the decoder (last linear) layer, or the norm of the embedding matrix. In both cases, the update magnitude is approximately proportional to the frequency of word usage, allowing for a greedy estimation by adapting the strategy of Wainakh et al (2021) which was originally proposed to extract label information in classification tasks.…”

Section: Getting Tokensmentioning

confidence: 99%

“…Algorithm 2 and Algorithm 3 detail the token recovery for transformers with decoder bias and for transformer with a tied embedding. These roughly follow the principles of greedy label recovery strategy proposed in Wainakh et al (2021) and we reproduce them here for completeness, incorporating additional considerations necessary for token retrieval.…”

Section: B Algorithm Detailsmentioning

confidence: 99%

Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Models

Fowl¹,

Geiping²,

Reich³

et al. 2022

Preprint

View full text Add to dashboard Cite

A central tenet of Federated learning (FL), which trains models without centralizing user data, is privacy. However, previous work has shown that the gradient updates used in FL can leak user information. While the most industrial uses of FL are for text applications (e.g. keystroke prediction), nearly all attacks on FL privacy have focused on simple image classifiers. We propose a novel attack that reveals private user text by deploying malicious parameter vectors, and which succeeds even with mini-batches, multiple users, and long sequences. Unlike previous attacks on FL, the attack exploits characteristics of both the Transformer architecture and the token embedding, separately extracting tokens and positional embeddings to retrieve high-fidelity text. This work suggests that FL on text, which has historically been resistant to privacy attacks, is far more vulnerable than previously thought. * Authors contributed equally. Order chosen randomly.

show abstract

User-Level Label Leakage from Gradients in Federated Learning

Cited by 10 publications

References 0 publications

Fishing for User Data in Large-Batch Federated Learning via Gradient Magnification

Fishing for User Data in Large-Batch Federated Learning via Gradient Magnification

Reconstructing Training Data with Informed Adversaries

Decepticons: Corrupted Transformers Breach Privacy in Federated Learning for Language Models

Contact Info

Product

Resources

About