The vulnerability of learning to adversarial perturbation increases with intrinsic dimensionality

Amsaleg, Laurent; Bailey, James E.; Barbe, Dominique; Erfani, Sarah M.; Houle, Michael E.; Nguyen, Vinh Duc; Radovanović, Miloš

doi:10.1109/wifs.2017.8267651

Cited by 42 publications

(49 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Defenses: Proposed defenses include detection and rejection methods [32,26,55,61,3,63], pre-processing, quantization and dimensionality reduction methods [12,73,7], manifold-projection methods [40,72,82,86], methods based on stochasticity/regularization or adapted architectures [109,7,68,88,35,43,76,45,51,107], ensemble methods [57,94,34,100], as well as adversarial training [109,65,36,83,90,54,62]; however, many defenses have been broken, often by considering "specialized" or novel attacks [13,15,5,6]. In [6], only adversarial training, e.g., the work by Madry et al [62], has been shown to be effective -although many recent defenses have not been studied extensively.…”

Section: Related Workmentioning

confidence: 99%

Disentangling Adversarial Robustness and Generalization

Stutz

Hein

Schiele

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

196

153

View full text Add to dashboard Cite

Obtaining deep networks that are robust against adversarial examples and generalize well is an open problem. A recent hypothesis [102,95] even states that both robust and accurate models are impossible, i.e., adversarial robustness and generalization are conflicting goals. In an effort to clarify the relationship between robustness and generalization, we assume an underlying, low-dimensional data manifold and show that: 1. regular adversarial examples leave the manifold; 2. adversarial examples constrained to the manifold, i.e., on-manifold adversarial examples, exist; 3. onmanifold adversarial examples are generalization errors, and on-manifold adversarial training boosts generalization; 4. regular robustness and generalization are not necessarily contradicting goals. These assumptions imply that both robust and accurate models are possible. However, different models (architectures, training strategies etc.) can exhibit different robustness and generalization characteristics. To confirm our claims, we present extensive experiments on synthetic data (with known manifold) as well as on EMNIST [19], Fashion-MNIST [106] and CelebA [59].

show abstract

Section: Related Workmentioning

confidence: 99%

Disentangling Adversarial Robustness and Generalization

Stutz

Hein

Schiele

2019

2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

196

153

View full text Add to dashboard Cite

show abstract

“…In a preliminary version of this paper, in the context of Euclidean spaces, we provided a theoretical explanation of the adversarial effect of perturbation for the closer + query scenario [28], in terms of the Local Intrinsic Dimensionality (LID) [29]- [31]. The LID characterizes the order of magnitude of the growth of probability measure with respect to a neighborhood of increasing radius.…”

Section: B Contributionsmentioning

confidence: 99%

High Intrinsic Dimensionality Facilitates Adversarial Attack: Theoretical Evidence

Amsaleg

Bailey

Barbe

et al. 2021

IEEE Trans.Inform.Forensic Secur.

Self Cite

View full text Add to dashboard Cite

Machine learning systems are vulnerable to adversarial attack. By applying to the input object a small, carefully-designed perturbation, a classifier can be tricked into making an incorrect prediction. This phenomenon has drawn wide interest, with many attempts made to explain it. However, a complete understanding is yet to emerge. In this paper we adopt a slightly different perspective, still relevant to classification. We consider retrieval, where the output is a set of objects most similar to a user-supplied query object, corresponding to the set of k-nearest neighbors. We investigate the effect of adversarial perturbation on the ranking of objects with respect to a query. Through theoretical analysis, supported by experiments, we demonstrate that as the intrinsic dimensionality of the data domain rises, the amount of perturbation required to subvert neighborhood rankings diminishes, and the vulnerability to adversarial attack rises. We examine two modes of perturbation of the query: either 'closer' to the target point, or 'farther' from it. We also consider two perspectives: 'query-centric', examining the effect of perturbation on the query's own neighborhood ranking, and 'target-centric', considering the ranking of the query point in the target's neighborhood set. All four cases correspond to practical scenarios involving classification and retrieval.

show abstract

“…First, phenomenological models are highly environment-specific, and any changes of the original field environment may result in a significant different outcome. Thus, they can show a considerable degree of stiffness in the prediction error, when not optimally trained against adversarial perturbations [23]. This means that in some cases the predictive errors may not have the smoothness the concept of applicability requires.…”

Section: Phenomenological Modelsmentioning

confidence: 99%

Credibility of In Silico Trial Technologies—A Theoretical Framing

Viceconti

Juárez

Curreli

et al. 2020

IEEE J. Biomed. Health Inform.

View full text Add to dashboard Cite

Different research communities have developed various approaches to assess the credibility of predictive models. Each approach usually works well for a specific type of model, and under some epistemic conditions that are normally satisfied within that specific research domain. Some regulatory agencies recently started to consider evidences of safety and efficacy on new medical products obtained using computer modelling and simulation (which is referred to as In Silico Trials); this has raised the attention in the computational medicine research community on the regulatory science aspects of this emerging discipline. But this poses a foundational problem: in the domain of biomedical research the use of computer modelling is relatively recent, without a widely accepted epistemic framing for model credibility. Also, because of the inherent complexity of living organisms, biomedical modellers tend to use a variety of modelling methods, sometimes mixing them in the solution of a single problem. In such context merely adopting credibility approaches developed within other research communities might not be appropriate. In this paper we propose a theoretical framing for assessing the credibility of a predictive models for In Silico Trials, which accounts for the epistemic specificity of this research field and is general enough to be used for different type of models.

show abstract

The vulnerability of learning to adversarial perturbation increases with intrinsic dimensionality

Cited by 42 publications

References 25 publications

Disentangling Adversarial Robustness and Generalization

Disentangling Adversarial Robustness and Generalization

High Intrinsic Dimensionality Facilitates Adversarial Attack: Theoretical Evidence

Credibility of In Silico Trial Technologies—A Theoretical Framing

Contact Info

Product

Resources

About