The Curse of Concentration in Robust Learning: Evasion and Poisoning Attacks from Concentration of Measure

Abstract

Many modern machine learning classifiers are shown to be vulnerable to adversarial perturbations of the instances. Despite a massive amount of work focusing on making classifiers robust, the task seems quite challenging. In this work, through a theoretical study, we investigate the adversarial risk and robustness of classifiers and draw a connection to the well-known phenomenon of "concentration of measure" in metric measure spaces. We show that if the metric probability space of the test instance is concentrated, any classifier with some initial constant error is inherently vulnerable to adversarial perturbations.One class of concentrated metric probability spaces are the so-called Lévy families that include many natural distributions. In this special case, our attacks only need to perturb the test instance by at most O( √ n) to make it misclassified, where n is the data dimension. Using our general result about Lévy instance spaces, we first recover as special case some of the previously proved results about the existence of adversarial examples. However, many more Lévy families are known (e.g., product distribution under the Hamming distance) for which we immediately obtain new attacks that find adversarial examples of distance O( √ n).Finally, we show that concentration of measure for product spaces implies the existence of forms of "poisoning" attacks in which the adversary tampers with the training data with the goal of degrading the classifier. In particular, we show that for any learning algorithm that uses m training examples, there is an adversary who can increase the probability of any "bad property" (e.g., failing on a particular test instance) that initially happens with 1/poly(m) probability to ≈ 1 by substituting only O( √ m) of the examples with other (still correctly labeled) examples.* This is the full version of a work appearing in AAAI 2019.

High Intrinsic Dimensionality Facilitates Adversarial Attack: Theoretical Evidence

“…The setup holds asymptotically with respect to the representational dimension. In the context of metric probability spaces, Mahloujifar et al [33] blend the aforementioned approaches, by arguing that the isoperimetric inequalities hold approximately over a much larger distribution family as the dimension m goes to infinity.…”

Computational Concentration of Measure: Optimal Bounds, Reductions, and More

“…In such so-called evasion attacks [BFR14, CW17, SZS + 14, GMP18] that find "adversarial examples", the goal of the adversary is to perturb the test input x into a "close" input x under some metric d (perhaps because this small perturbation is imperceptible to humans) in such a way that this tampering makes the hypothesis h make a mistake. In [MDM19], it was also shown that the concentration of measure can potentially lead to inherent evasion attacks, as long as the input metric probability space (X , d, µ) is concentrated. This holds e.g., if the space is a Normal Lévy family [Lév51,AM85].…”

Characterizing the Shape of Activation Space in Deep Neural Networks

“…Most notably, Katz et al [9] and Weng et al [21] have found that computing a provably secure region of the input space is approximately computationally hard. Mahloujifar et al [14] explain the prevalence of adversarial examples by making a connection to the "concentration of measure" in metric spaces. Recently, Zhange et al [23] found that adversaries are more dense sufficiently far from the manifold of training data.…”