Limitations of the Lipschitz constant as a defense against adversarial examples

Huster, Todd; Chiang, Cho‐Yu Jason; Chadha, Ritu

doi:10.48550/arxiv.1807.09705

Cited by 4 publications

(10 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…However, all traditional networks following this approach are greatly limited; Huster showed that no traditional neural network such as those using the ReLU activation function can act as a universal Lipschitz approximator. Traditional networks must in fact choose between either the expressive power necessary to approximate a function, or the Lipschitz condition [12]. The search continued into non-traditional neural networks, and in 2018 Anil and Lucas showed that by changing out a standard monotonically increasing activation function for a sorting activation function, their networks, given arbitrary depth, form a ULA [1].…”

Section: Related Workmentioning

confidence: 99%

“…For example, as we alluded to earlier, all differentiable functions with bounded derivative are Lipschitz continuous. Perhaps more importantly, given any finite dataset where different classes are separated in the input space by at least a distance of c, there exists a Lipschitz function with Lipschitz constant c/2 that correctly classifies all points [12]. Therefore, any classification problem can be restated in the structure of a Lipschitz function approximation problem.…”

Section: Definitionsmentioning

confidence: 99%

“…Thus, these networks can only act as Lipschitz approximators on a very limited class of functions. A class of functions so limited that it does not even include the absolute value function [12]! Anil and Lucas suggest that this is due to the norm-decreasing properties of these traditional monotonically increasing activation functions.…”

Section: Definitionsmentioning

confidence: 99%

“…Compounding the problesm, empirically effective defenses to model robustness have often been broken soon after their invention [2,3]. This is due to the fundamental fact that attacks can always take advantage of large gradients that traditional neural networks must have in order to represent their target functions [12]. Lipschitz approximation avoids this problem by design.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Universal Lipschitz Approximation in Bounded Depth Neural Networks

Cohen,

Huster,

Cohen

2019

Preprint

Self Cite

View full text Add to dashboard Cite

Adversarial attacks against machine learning models are a rather hefty obstacle to our increasing reliance on these models. Due to this, provably robust (certified) machine learning models are a major topic of interest. Lipschitz continuous models present a promising approach to solving this problem. By leveraging the expressive power of a variant of neural networks which maintain low Lipschitz constants, we prove that three layer neural networks using the FullSort activation function are Universal Lipschitz function Approximators (ULAs). This both explains experimental results and paves the way for the creation of better certified models going forward. We conclude by presenting experimental results that suggest that ULAs are a not just a novelty, but a competitive approach to providing certified classifiers, using these results to motivate several potential topics of further research.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: Definitionsmentioning

confidence: 99%

Section: Definitionsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Universal Lipschitz Approximation in Bounded Depth Neural Networks

Cohen,

Huster,

Cohen

2019

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…The Lipschitz constant is utilized to bound DNNs' vulnerability to adversarial attacks [9,10]. As argued in [11,12], however, currently there is no accurate method for estimating the Lipschitz constant, and the resulting overestimation can easily render its use unpractical. [13,14] propose to train a generative model for generating unseen samples for which misclassification happens.…”

Section: Introductionmentioning

confidence: 99%

Global Adversarial Attacks for Assessing Deep Learning Robustness

Hu¹,

Shah²,

Huang³

2019

Preprint

View full text Add to dashboard Cite

It has been shown that deep neural networks (DNNs) may be vulnerable to adversarial attacks, raising the concern on their robustness particularly for safety-critical applications. Recognizing the local nature and limitations of existing adversarial attacks, we present a new type of global adversarial attacks for assessing global DNN robustness. More specifically, we propose a novel concept of global adversarial example pairs in which each pair of two examples are close to each other but have different class labels predicted by the DNN. We further propose two families of global attack methods and show that our methods are able to generate diverse and intriguing adversarial example pairs at locations far from the training or testing data. Moreover, we demonstrate that DNNs hardened using the strong projected gradient descent (PGD) based (local) adversarial training are vulnerable to the proposed global adversarial example pairs, suggesting that global robustness must be considered while training robust deep learning networks.Preprint. Under review.

show abstract

Robust and provably monotonic networks

Kitouni,

Nolte,

Williams

2023

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

The Lipschitz constant of the map between the input and output space represented by a neural network is a natural metric for assessing the robustness of the model. We present a new method to constrain the Lipschitz constant of dense deep learning models that can also be generalized to other architectures. The method relies on a simple weight normalization scheme during training that ensures the Lipschitz constant of every layer is below an upper limit specified by the analyst. A simple monotonic residual connection can then be used to make the model monotonic in any subset of its inputs, which is useful in scenarios where domain knowledge dictates such dependence. Examples can be found in algorithmic fairness requirements or, as presented here, in the classification of the decays of subatomic particles produced at the CERN Large Hadron Collider. Our normalization is minimally constraining and allows the underlying architecture to maintain higher expressiveness compared to other techniques which aim to either control the Lipschitz constant of the model or ensure its monotonicity. We show how the algorithm was used to train a powerful, robust, and interpretable discriminator for heavy-flavor-quark decays, which has been adopted for use as the primary data-selection algorithm in the LHCb real-time data-processing system in the current LHC data-taking period known as Run~3.In addition, our algorithm has also achieved state-of-the-art performance on benchmarks in medicine, finance, and other applications.

show abstract

Limitations of the Lipschitz constant as a defense against adversarial examples

Cited by 4 publications

References 0 publications

Universal Lipschitz Approximation in Bounded Depth Neural Networks

Universal Lipschitz Approximation in Bounded Depth Neural Networks

Global Adversarial Attacks for Assessing Deep Learning Robustness

Robust and provably monotonic networks

Contact Info

Product

Resources

About