On Lipschitz Bounds of General Convolutional Neural Networks

Zou, Dongmian; Bălan, Radu; Singh, Maneesh

doi:10.1109/tit.2019.2961812

Cited by 33 publications

(22 citation statements)

References 22 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(2018) propose a similar method for computing the spectral norm of a convolutional layer, with the intention of regularising it in order to improve the adversarial robustness of the resulting model. Zou et al (2019) propose a general framework for computing bounds on Lipschitz constants by solving a linear program.…”

Section: Related Workmentioning

confidence: 99%

Regularisation of neural networks by enforcing Lipschitz continuity

et al. 2020

View full text Add to dashboard Cite

We investigate the effect of explicitly enforcing the Lipschitz continuity of neural networks with respect to their inputs. To this end, we provide a simple technique for computing an upper bound to the Lipschitz constant—for multiple p-norms—of a feed forward neural network composed of commonly used layer types. Our technique is then used to formulate training a neural network with a bounded Lipschitz constant as a constrained optimisation problem that can be solved using projected stochastic gradient methods. Our evaluation study shows that the performance of the resulting models exceeds that of models trained with other common regularisers. We also provide evidence that the hyperparameters are intuitive to tune, demonstrate how the choice of norm for computing the Lipschitz constant impacts the resulting model, and show that the performance gains provided by our method are particularly noticeable when only a small amount of training data is available.

show abstract

Section: Related Workmentioning

confidence: 99%

Regularisation of neural networks by enforcing Lipschitz continuity

et al. 2020

View full text Add to dashboard Cite

show abstract

“…A semidefinite programming technique (LipSDP) is presented in [8] to compute Lipschitz bounds, but in order to apply it to larger networks, a relaxation must be used which invalidates the guarantee. Another approach is that of [9], in which linear programming is used to estimate Lipschitz constants. The downside to all of these approaches is that they usually can only be applied to small networks, and also often have to be relaxed, which invalidates any guarantee on the bound.…”

Section: Introductionmentioning

confidence: 99%

Analytical bounds on the local Lipschitz constants of ReLU networks

Avant¹,

Morgansen²

2021

Preprint

View full text Add to dashboard Cite

In this paper, we determine analytical upper bounds on the local Lipschitz constants of feedforward neural networks with ReLU activation functions. We do so by deriving Lipschitz constants and bounds for ReLU, affine-ReLU, and max pooling functions, and combining the results to determine a networkwide bound. Our method uses several insights to obtain tight bounds, such as keeping track of the zero elements of each layer, and analyzing the composition of affine and ReLU functions. Furthermore, we employ a careful computational approach which allows us to apply our method to large networks such as AlexNet and VGG-16. We present several examples using different networks, which show how our local Lipschitz bounds are tighter than the global Lipschitz bounds. We also show how our method can be applied to provide adversarial bounds for classification networks. These results show that our method produces the largest known bounds on minimum adversarial perturbations for large networks such as AlexNet and VGG-16.

show abstract

“…To obtain a better understanding of learning algorithms, the Lipschitz distribution is widely used for a theoretical analysis of neural networks. For instance, in CNNs (Zou, Balan, and Singh 2018), the Lipschitz bound is important in the study of the stability and the computation of the Lipschitz bound is used for generative networks. In (Zou, Balan, and Singh 2018) the authors give a general framework for CNNs and prove that the Lipschitz bound of a CNN can be determined by solving a linear program with a more explicit expression for a suboptimal bound.…”

Section: Implementation Of the Gradient Calibration Layermentioning

confidence: 99%

“…For instance, in CNNs (Zou, Balan, and Singh 2018), the Lipschitz bound is important in the study of the stability and the computation of the Lipschitz bound is used for generative networks. In (Zou, Balan, and Singh 2018) the authors give a general framework for CNNs and prove that the Lipschitz bound of a CNN can be determined by solving a linear program with a more explicit expression for a suboptimal bound. In light of this, we theoretically show that an unbiased variable estimator can be achieved in CSGD with a Lipschitz assumption in a probabilistic way.…”

Section: Implementation Of the Gradient Calibration Layermentioning

confidence: 99%

“…The training procedure is terminated at 64k iterations, which is determined based on a 45k/5k train/validation split. We follow the same data augmentation strategy in (Zou, Balan, and Singh 2018) for training. Horizontal flipping is adopted, and a 32 × 32 crop is sampled randomly from the image padded with 4 pixels on each side.…”

Section: Experiments On Natural Image Classificationmentioning

confidence: 99%

See 1 more Smart Citation

Calibrated Stochastic Gradient Descent for Convolutional Neural Networks

Zhuo

Zhang

Chen

et al. 2019

AAAI

View full text Add to dashboard Cite

In stochastic gradient descent (SGD) and its variants, the optimized gradient estimators may be as expensive to compute as the true gradient in many scenarios. This paper introduces a calibrated stochastic gradient descent (CSGD) algorithm for deep neural network optimization. A theorem is developed to prove that an unbiased estimator for the network variables can be obtained in a probabilistic way based on the Lipschitz hypothesis. Our work is significantly distinct from existing gradient optimization methods, by providing a theoretical framework for unbiased variable estimation in the deep learning paradigm to optimize the model parameter calculation. In particular, we develop a generic gradient calibration layer which can be easily used to build convolutional neural networks (CNNs). Experimental results demonstrate that CNNs with our CSGD optimization scheme can improve the stateof-the-art performance for natural image classification, digit recognition, ImageNet object classification, and object detection tasks. This work opens new research directions for developing more efficient SGD updates and analyzing the backpropagation algorithm.

show abstract

On Lipschitz Bounds of General Convolutional Neural Networks

Cited by 33 publications

References 22 publications

Regularisation of neural networks by enforcing Lipschitz continuity

Regularisation of neural networks by enforcing Lipschitz continuity

Analytical bounds on the local Lipschitz constants of ReLU networks

Calibrated Stochastic Gradient Descent for Convolutional Neural Networks

Contact Info

Product

Resources

About