A Methodology for Formalizing Model-Inversion Attacks

Wu, Xi; Fredrikson, Matthew; Jha, Somesh; Naughton, Jeffrey F.

doi:10.1109/csf.2016.32

Cited by 143 publications

(110 citation statements)

References 15 publications

Supporting

Mentioning

109

Contrasting

Unclassified

Order By: Relevance

“…Machine learning has emerged as an important technology, enabling a wide range of applications including computer vision, machine translation, health analytics, and advertising, among others. The fact that many compelling applications of this technology involve the collection and processing of sensitive personal data has given rise to concerns about privacy [1,2,3,4,5,6,7,8,9]. In particular, when machine learning algorithms are applied to private training data, the resulting models might unwittingly leak information about that data through either their behavior (i.e., black-box attack) or the details of their structure (i.e., white-box attack).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting

Yeom

Giacomelli

Fredrikson

et al. 2018

2018 IEEE 31st Computer Security Foundations Symposium (CSF)

Self Cite

671

787

View full text Add to dashboard Cite

Machine learning algorithms, when applied to sensitive data, pose a distinct threat to privacy. A growing body of prior work demonstrates that models produced by these algorithms may leak specific private information in the training data to an attacker, either through the models' structure or their observable behavior. However, the underlying cause of this privacy risk is not well understood beyond a handful of anecdotal accounts that suggest overfitting and influence might play a role.This paper examines the effect that overfitting and influence have on the ability of an attacker to learn information about the training data from machine learning models, either through training set membership inference or attribute inference attacks. Using both formal and empirical analyses, we illustrate a clear relationship between these factors and the privacy risk that arises in several popular machine learning algorithms. We find that overfitting is sufficient to allow an attacker to perform membership inference and, when the target attribute meets certain conditions about its influence, attribute inference attacks. Interestingly, our formal analysis also shows that overfitting is not necessary for these attacks and begins to shed light on what other factors may be in play. Finally, we explore the connection between membership inference and attribute inference, showing that there are deep connections between the two that lead to effective new attacks. * This is the unabridged version of the paper accepted for publication in CSF 2018.

show abstract

Section: Introductionmentioning

confidence: 99%

“…A second factor identified as relevant to privacy risk is influence [5], a quantity that arises often in the study of Boolean functions [20]. Influence measures the extent to which a particular input to a function is able to cause changes to its output.…”

Section: Introductionmentioning

confidence: 99%

Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting

Yeom

Giacomelli

Fredrikson

et al. 2018

2018 IEEE 31st Computer Security Foundations Symposium (CSF)

Self Cite

671

787

View full text Add to dashboard Cite

show abstract

“…In this attack, the attacker attempts to mimic the data samples to deceive the ML algorithms to classifying the original samples with different labels incorrectly from the impersonated ones [272,274,275]. The last possible attack is inversion, which exploits the application program interfaces presented to the users by the current ML platform to collect roughly the necessary information about the pre-trained ML models [271,276]. Subsequently, this extracted information is used to perform reverse engineering to obtain the sensitive data of users.…”

Section: ) Security Of ML and Dl Methodsmentioning

confidence: 99%

A Survey of Machine and Deep Learning Methods for Internet of Things (IoT) Security

Al-Garadi

Mohamed

Al-Ali

et al. 2020

IEEE Commun. Surv. Tutorials

838

333

View full text Add to dashboard Cite

The Internet of Things (IoT) integrates billions of smart devices that can communicate with one another with minimal human intervention. It is one of the fastest developing fields in the history of computing, with an estimated 50 billion devices by the end of 2020. On the one hand, IoT technologies play a crucial role in enhancing several real-life smart applications that can improve life quality. On the other hand, the crosscutting nature of IoT systems and the multidisciplinary components involved in the deployment of such systems have introduced new security challenges. Implementing security measures, such as encryption, authentication, access control, network security and application security, for IoT devices and their inherent vulnerabilities is ineffective. Therefore, existing security methods should be enhanced to secure the IoT ecosystem effectively. Machine learning and deep learning (ML/DL) have advanced considerably over the last few years, and machine intelligence has transitioned from laboratory curiosity to practical machinery in several important applications. The ability to monitor IoT devices intelligently provides a significant solution to new or zero-day attacks. ML/DL are powerful methods of data exploration for learning about 'normal' and 'abnormal' behaviour according to how IoT components and devices perform within the IoT environment. Consequently, ML/DL methods are important in transforming the security of IoT systems from merely facilitating secure communication between devices to security-based intelligence systems. The goal of this work is to provide a comprehensive survey of ML methods and recent advances in DL methods that can be used to develop enhanced security methods for IoT systems.IoT security threats that are related to inherent or newly introduced threats are presented, and various potential IoT system attack surfaces and the possible threats related to each surface are discussed. We then thoroughly review ML/DL methods for IoT security and present the opportunities, advantages and shortcomings of each method. We discuss the opportunities and challenges involved in applying ML/DL to IoT security. These opportunities and challenges can serve as potential future research directions.

show abstract

“…Data pollution attacks have been studied long before other attacks became relevant [27]. In a model inversion attack, the attacker learns information about data used to train the machine learning model [28], [29], [30]. A similar but stronger setting is the membership inference attack where the attacker identifies whether an individual's information was present in the training data [31].…”

Section: Commentsmentioning

confidence: 99%

Towards Understanding Limitations of Pixel Discretization Against Adversarial Attacks

Chen

Rastogi

et al. 2019

2019 IEEE European Symposium on Security and Privacy (EuroS&P)

Self Cite

View full text Add to dashboard Cite

Wide adoption of artificial neural networks in various domains has led to an increasing interest in defending adversarial attacks against them. Preprocessing defense methods such as pixel discretization are particularly attractive in practice due to their simplicity, low computational overhead, and applicability to various systems. It is observed that such methods work well on simple datasets like MNIST, but break on more complicated ones like ImageNet under recently proposed strong white-box attacks. To understand the conditions for success and potentials for improvement, we study the pixel discretization defense method, including more sophisticated variants that take into account the properties of the dataset being discretized. Our results again show poor resistance against the strong attacks. We analyze our results in a theoretical framework and offer strong evidence that pixel discretization is unlikely to work on all but the simplest of the datasets. Furthermore, our arguments present insights why some other preprocessing defenses may be insecure.

show abstract

A Methodology for Formalizing Model-Inversion Attacks

Cited by 143 publications

References 15 publications

Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting

Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting

A Survey of Machine and Deep Learning Methods for Internet of Things (IoT) Security

Towards Understanding Limitations of Pixel Discretization Against Adversarial Attacks

Contact Info

Product

Resources

About