Privacy Risks of Securing Machine Learning Models against Adversarial Examples

Song, Liwei; Shokri, Reza; Mittal, Prateek

doi:10.1145/3319535.3354211

Cited by 191 publications

(159 citation statements)

References 40 publications

Supporting

Mentioning

157

Contrasting

Order By: Relevance

“…In a concurrent work, Song et al [58] evaluate several different attacks that seek to extract membership information from robust models, showing that robustness can make a model more vulnerable to membership inference. Although the attacks that we present within our formal framework are similar to theirs, our experimental setup has a few major differences.…”

Section: Robustnessmentioning

confidence: 99%

Overfitting, robustness, and malicious algorithms: A study of potential causes of privacy risk in machine learning

Yeom

Giacomelli

Menaged

et al. 2020

JCS

View full text Add to dashboard Cite

Machine learning algorithms, when applied to sensitive data, pose a distinct threat to privacy. A growing body of prior work demonstrates that models produced by these algorithms may leak specific private information in the training data to an attacker, either through the models' structure or their observable behavior. This article examines the factors that can allow a training set membership inference attacker or an attribute inference attacker to learn such information. Using both formal and empirical analyses, we illustrate a clear relationship between these factors and the privacy risk that arises in several popular machine learning algorithms. We find that overfitting is sufficient to allow an attacker to perform membership inference and, when the target attribute meets certain conditions about its influence, attribute inference attacks. We also explore the connection between membership inference and attribute inference, showing that there are deep connections between the two that lead to effective new attacks. We show that overfitting is not necessary for these attacks, demonstrating that other factors, such as robustness to norm-bounded input perturbations and malicious training algorithms, can also significantly increase the privacy risk. Notably, as robustness is intended to be a defense against attacks on the integrity of model predictions, these results suggest it may be difficult in some cases to simultaneously defend against privacy and integrity attacks.

show abstract

Section: Robustnessmentioning

confidence: 99%

Overfitting, robustness, and malicious algorithms: A study of potential causes of privacy risk in machine learning

Yeom

Giacomelli

Menaged

et al. 2020

JCS

View full text Add to dashboard Cite

show abstract

“…There have been some recent works in analyzing the connection between membership inference and adversarial machine learning. Song et al [48] and Mejia et. al.…”

Section: Related Workmentioning

confidence: 96%

Effects of Differential Privacy and Data Skewness on Membership Inference Vulnerability

Truex

Liu

Gürsoy

et al. 2019

2019 First IEEE International Conference on Trust, Privacy and Security in Intelligent Systems and Applications (TPS-ISA)

View full text Add to dashboard Cite

Membership inference attacks seek to infer the membership of individual training instances of a privately trained model. This paper presents a membership privacy analysis and evaluation system, called MPLens, with three unique contributions. First, through MPLens, we demonstrate how membership inference attack methods can be leveraged in adversarial machine learning. Second, through MPLens, we highlight how the vulnerability of pre-trained models under membership inference attack is not uniform across all classes, particularly when the training data itself is skewed. We show that risk from membership inference attacks is routinely increased when models use skewed training data. Finally, we investigate the effectiveness of differential privacy as a mitigation technique against membership inference attacks. We discuss the trade-offs of implementing such a mitigation strategy with respect to the model complexity, the learning task complexity, the dataset complexity and the privacy parameter settings. Our empirical results reveal that (1) minority groups within skewed datasets display increased risk for membership inference and (2) differential privacy presents many challenging trade-offs as a mitigation technique to membership inference risk.

show abstract

“…Finally, an approach by Nasr et al [47] consisted of using an adversarial algorithm to perform inference attacks against trained models. Song et al [48] proposed two new methods of exploiting the structural properties of adversarial conscious datasets, thus proving the investigated inference defences ineffective.…”

Section: Related Workmentioning

confidence: 99%

Towards a Multi-Layered Phishing Detection

Rendall

Nisioti

Mylonas

2020

Sensors

View full text Add to dashboard Cite

Phishing is one of the most common threats that users face while browsing the web. In the current threat landscape, a targeted phishing attack (i.e., spear phishing) often constitutes the first action of a threat actor during an intrusion campaign. To tackle this threat, many data-driven approaches have been proposed, which mostly rely on the use of supervised machine learning under a single-layer approach. However, such approaches are resource-demanding and, thus, their deployment in production environments is infeasible. Moreover, most previous works utilise a feature set that can be easily tampered with by adversaries. In this paper, we investigate the use of a multi-layered detection framework in which a potential phishing domain is classified multiple times by models using different feature sets. In our work, an additional classification takes place only when the initial one scores below a predefined confidence level, which is set by the system owner. We demonstrate our approach by implementing a two-layered detection system, which uses supervised machine learning to identify phishing attacks. We evaluate our system with a dataset consisting of active phishing attacks and find that its performance is comparable to the state of the art.

show abstract

Privacy Risks of Securing Machine Learning Models against Adversarial Examples

Cited by 191 publications

References 40 publications

Overfitting, robustness, and malicious algorithms: A study of potential causes of privacy risk in machine learning

Overfitting, robustness, and malicious algorithms: A study of potential causes of privacy risk in machine learning

Effects of Differential Privacy and Data Skewness on Membership Inference Vulnerability

Towards a Multi-Layered Phishing Detection

Contact Info

Product

Resources

About