Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning

Nasr, Milad; Shokri, Reza; Houmansadr, Amir

doi:10.1109/sp.2019.00065

Cited by 1,133 publications

(933 citation statements)

References 25 publications

Supporting

Mentioning

816

Contrasting

Unclassified

Order By: Relevance

“…We evaluate MemGuard and compare it with state-of-the-art defenses [1,42,56,58] on three real-world datasets. Our empirical results show that MemGuard can effectively defend against state-of-the-art black-box membership inference attacks [43,56]. In particular, as MemGuard is allowed to add larger noise (we measure the magnitude of the noise using its L 1 -norm), the inference accuracies of all evaluated membership inference attacks become smaller.…”

Section: Introductionmentioning

confidence: 93%

See 1 more Smart Citation

MemGuard

Jia

Salem

Backes

et al. 2019

Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security

240

View full text Add to dashboard Cite

In a membership inference attack, an attacker aims to infer whether a data sample is in a target classifier's training dataset or not. Specifically, given a black-box access to the target classifier, the attacker trains a binary classifier, which takes a data sample's confidence score vector predicted by the target classifier as an input and predicts the data sample to be a member or non-member of the target classifier's training dataset. Membership inference attacks pose severe privacy and security threats to the training dataset. Most existing defenses leverage differential privacy when training the target classifier or regularize the training process of the target classifier. These defenses suffer from two key limitations: 1) they do not have formal utility-loss guarantees of the confidence score vectors, and 2) they achieve suboptimal privacy-utility tradeoffs.In this work, we propose MemGuard, the first defense with formal utility-loss guarantees against black-box membership inference attacks. Instead of tampering the training process of the target classifier, MemGuard adds noise to each confidence score vector predicted by the target classifier. Our key observation is that attacker uses a classifier to predict member or non-member and classifier is vulnerable to adversarial examples. Based on the observation, we propose to add a carefully crafted noise vector to a confidence score vector to turn it into an adversarial example that misleads the attacker's classifier. Specifically, MemGuard works in two phases. In Phase I, MemGuard finds a carefully crafted noise vector that can turn a confidence score vector into an adversarial example, which is likely to mislead the attacker's classifier to make a random guessing at member or non-member. We find such carefully crafted noise vector via a new method that we design to incorporate the unique utility-loss constraints on the noise vector. In Phase II, Mem-Guard adds the noise vector to the confidence score vector with a certain probability, which is selected to satisfy a given utility-loss budget on the confidence score vector. Our experimental results on Permission to make digital three datasets show that MemGuard can effectively defend against membership inference attacks and achieve better privacy-utility tradeoffs than existing defenses. Our work is the first one to show that adversarial examples can be used as defensive mechanisms to defend against membership inference attacks.

show abstract

Section: Introductionmentioning

confidence: 93%

“…More recently, Nasr et al [43] proposed membership inference attacks against white-box ML models. For a data sample, they calculate the corresponding gradients over the white-box target classifier's parameters and use these gradients as the data sample's feature for membership inference.…”

Section: Related Work 21 Membership Inferencementioning

confidence: 99%

MemGuard

Jia

Salem

Backes

et al. 2019

Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security

240

View full text Add to dashboard Cite

show abstract

“…However, [39], [41], [42] failed to present performance analysis of the state-of-the-art edge caching schemes. On the other hand, [43] compared the performance of the FL and centralized schemes. However, the analysis was limited to the privacy performance of the schemes.…”

Section: B Literature Reviewmentioning

confidence: 99%

A Classification of the Enabling Techniques for Low Latency and Reliable Communications in 5G and Beyond: AI-Enabled Edge Caching

Mutalemwa

Shin

2020

IEEE Access

View full text Add to dashboard Cite

“…Membership inference. Membership inference against classification models has been studied in [22,25,27,28], and later studied for generative models and language models [15,31] as well as in collaborative learning setting [23,24]. The attack in [28] focuses black-box models, exploiting the differences in the models' outputs on training and non-training inputs.…”

Section: Related Workmentioning

confidence: 99%

Membership Encoding for Deep Learning

Song

Shokri

2020

Proceedings of the 15th ACM Asia Conference on Computer and Communications Security

Self Cite

View full text Add to dashboard Cite

Machine learning as a service (MLaaS), and algorithm marketplaces are on a rise. Data holders can easily train complex models on their data using third party provided learning codes. Training accurate ML models requires massive labeled data and advanced learning algorithms. The resulting models are considered as intellectual property of the model owners and their copyright should be protected. Also, MLaaS needs to be trusted not to embed secret information about the training data into the model, such that it could be later retrieved when the model is deployed. In this paper, we present membership encoding for training deep neural networks and encoding the membership information, i.e. whether a data point is used for training, for a subset of training data. Membership encoding has several applications in different scenarios, including robust watermarking for model copyright protection, and also the risk analysis of stealthy data embedding privacy attacks. Our encoding algorithm can determine the membership of significantly redacted data points, and is also robust to model compression and fine-tuning. It also enables encoding a significant fraction of the training set, with negligible drop in the model's prediction accuracy. CCS CONCEPTS • Computing methodologies → Machine learning; • Security and privacy → Software and application security.

show abstract

Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning

Cited by 1,133 publications

References 25 publications

MemGuard

MemGuard

A Classification of the Enabling Techniques for Low Latency and Reliable Communications in 5G and Beyond: AI-Enabled Edge Caching

Membership Encoding for Deep Learning

Contact Info

Product

Resources

About