Security and Machine Learning in the Real World

Evtimov, Ivan; Cui, Weidong; Kamar, Ece; Kıcıman, Emre; Kohno, Tadayoshi; Li, Jerry

doi:10.48550/arxiv.2007.07205

Cited by 6 publications

(9 citation statements)

References 29 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…As such, the computed perturbations are confined to the digital domain, and are impossible to be deployed to a physical target. Recently, there has been a surging research interest on the performance of these attacks when deployed on real-world systems [11]. Along these lines, in our work, we develop perturbations that are designed to be deployable.…”

Section: Patchmentioning

confidence: 99%

“…These techniques encompass robust training procedures [23], perturbation detection and removal [14], and reconstructions through deep image priors [9]. To defend from Adversarial Scratches, following [11], we consider defenses that rely on input filtering as these are more scalable than defenses that try to make the model itself more robust. In particular, we adopt:…”

Section: Defensesmentioning

confidence: 99%

See 1 more Smart Citation

Adversarial Scratches: Deployable Attacks to CNN Classifiers

Giulivi,

Jere,

Rossi

et al. 2022

Preprint

View full text Add to dashboard Cite

A growing body of work has shown that deep neural networks are susceptible to adversarial examples. These take the form of small perturbations applied to the model's input which lead to incorrect predictions. Unfortunately, most literature focuses on visually imperceivable perturbations to be applied to digital images that often are, by design, impossible to be deployed to physical targets.We present Adversarial Scratches: a novel L 0 black-box attack, which takes the form of scratches in images, and which possesses much greater deployability than other state-of-the-art attacks. Adversarial Scratches leverage Bézier Curves to reduce the dimension of the search space and possibly constrain the attack to a specific location.We test Adversarial Scratches in several scenarios, including a publicly available API and images of traffic signs. Results show that, often, our attack achieves higher fooling rate than other deployable state-of-the-art methods, while requiring significantly fewer queries and modifying very few pixels.

show abstract

Section: Patchmentioning

confidence: 99%

Section: Defensesmentioning

confidence: 99%

Adversarial Scratches: Deployable Attacks to CNN Classifiers

Giulivi,

Jere,

Rossi

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…In general, AML research has been criticized for the limited practical relevance of its threat models [28,31]. There is also limited knowledge about which threats are relevant in practice.…”

Section: Adversarial Machine Learningmentioning

confidence: 99%

Mental Models of Adversarial Machine Learning

Bieringer¹,

Grosse²,

Backes³

et al. 2021

Preprint

View full text Add to dashboard Cite

Although machine learning (ML) is widely used in practice, little is known about practitioners' actual understanding of potential security challenges. In this work, we close this substantial gap in the literature and contribute a qualitative study focusing on developers' mental models of the ML pipeline and potentially vulnerable components. Studying mental models has helped in other security fields to discover root causes or improve risk communication. Our study reveals four characteristic ranges in mental models of industrial practitioners. The first range concerns the intertwined relationship of adversarial machine learning (AML) and classical security. The second range describes structural and functional components. The third range expresses individual variations of mental models, which are neither explained by the application nor by the educational background of the corresponding subjects. The fourth range corresponds to the varying levels of technical depth, which are however not determined by our subjects' level of knowledge. Our characteristic ranges have implications for the integration of AML into corporate workflows, security enhancing tools for practitioners, and creating appropriate regulatory frameworks for AML.

show abstract

“…Second, in deployed systems, a ML model typically interacts with other components, including other models. This interaction can be of extreme complexity, which might introduce additional challenges for adversaries [34]. For instance, in the Gboard app [48], as a user starts typing a search query, a baseline model determines possible search suggestions.…”

Section: Fallacies In Evaluation Setupsmentioning

confidence: 99%

“…Description. An attack becomes ineffective if it requires the adversary to make a disproportional large effort to overcome a small defense mechanism [34]. Proposed attacks need to be evaluated in this respect with stateof-the-art defenses.…”

Section: Fallacies In Evaluation Setupsmentioning

confidence: 99%

Federated Learning Attacks Revisited: A Critical Discussion of Gaps, Assumptions, and Evaluation Setups

Wainakh¹,

Zimmer²,

Subedi³

et al. 2021

Preprint

View full text Add to dashboard Cite

Federated learning (FL) enables a set of entities to collaboratively train a machine learning model without sharing their sensitive data, thus, mitigating some privacy concerns. However, an increasing number of works in the literature propose attacks that can manipulate the model and disclose information about the training data in FL. As a result, there has been a growing belief in the research community that FL is highly vulnerable to a variety of severe attacks. Although these attacks do indeed highlight security and privacy risks in FL, some of them may not be as effective in production deployment because they are feasible only under special-sometimes impractical-assumptions. Furthermore, some attacks are evaluated under limited setups that may not match real-world scenarios. In this paper, we investigate this issue by conducting a systematic mapping study of attacks against FL, covering 48 relevant papers from 2016 to the third quarter of 2021. On the basis of this study, we provide a quantitative analysis of the proposed attacks and their evaluation settings. This analysis reveals several research gaps with regard to the type of target ML models and their architectures. Additionally, we highlight unrealistic assumptions in the problem settings of some attacks, related to the hyper-parameters of the ML model and data distribution among clients. Furthermore, we identify and discuss several fallacies in the evaluation of attacks, which open up questions on the generalizability of the conclusions. As a remedy, we propose a set of recommendations to avoid these fallacies and to promote adequate evaluations.

show abstract

Security and Machine Learning in the Real World

Cited by 6 publications

References 29 publications

Adversarial Scratches: Deployable Attacks to CNN Classifiers

Adversarial Scratches: Deployable Attacks to CNN Classifiers

Mental Models of Adversarial Machine Learning

Federated Learning Attacks Revisited: A Critical Discussion of Gaps, Assumptions, and Evaluation Setups

Contact Info

Product

Resources

About