Gradient Regularization Improves Accuracy of Discriminative Models

Varga, Dániel; Csiszárik, Adrián; Zombori, Zsolt

doi:10.48550/arxiv.1712.09936

Cited by 14 publications

(21 citation statements)

References 9 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When considering such an approximate algorithm, one naively must trade off efficiency against accuracy for computing the Jacobian, which ultimately trades computation time for robustness. Prior work by Varga et al [2017] briefly considers an approach based on random projection, but without providing any analysis on the quality of the Jacobian approximation. Here, we describe our algorithm, analyze theoretical convergence guarantees, and verify empirically that there is only a negligible difference in model solution quality between training with the exact computation of the Jacobian as compared to training with the approximate algorithm, even when using a single random projection (see Figure 2).…”

Section: Efficient Approximate Algorithmmentioning

confidence: 99%

Robust Learning with Jacobian Regularization

Hoffman¹,

Roberts²,

Yaida³

2019

Preprint

View full text Add to dashboard Cite

Design of reliable systems must guarantee stability against input perturbations. In machine learning, such guarantee entails preventing overfitting and ensuring robustness of models against corruption of input data. In order to maximize stability, we analyze and develop a computationally efficient implementation of Jacobian regularization that increases classification margins of neural networks. The stabilizing effect of the Jacobian regularizer leads to significant improvements in robustness, as measured against both random and adversarial input perturbations, without severely degrading generalization properties on clean data.

show abstract

Section: Efficient Approximate Algorithmmentioning

confidence: 99%

Robust Learning with Jacobian Regularization

Hoffman¹,

Roberts²,

Yaida³

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Finally our work is in line with recent research (Liu et al, 2019a;Santurkar et al, 2018) that emphasizes the benefit of analyzing gradients to understand neural networks and devise potential improvements to their training. We share elements with Drucker & Le Cun (1991) and more recently Varga et al (2017) in that we propose explicit regularization methods for gradients.…”

Section: Related Workmentioning

confidence: 99%

Regularizing Deep Multi-Task Networks using Orthogonal Gradients

Suteu,

Guo

2019

Preprint

View full text Add to dashboard Cite

Deep neural networks are a promising approach towards multi-task learning because of their capability to leverage knowledge across domains and learn general purpose representations. Nevertheless, they can fail to live up to these promises as tasks often compete for a model's limited resources, potentially leading to lower overall performance. In this work we tackle the issue of interfering tasks through a comprehensive analysis of their training, derived from looking at the interaction between gradients within their shared parameters. Our empirical results show that well-performing models have low variance in the angles between task gradients and that popular regularization methods implicitly reduce this measure. Based on this observation, we propose a novel gradient regularization term that minimizes task interference by enforcing near orthogonal gradients. Updating the shared parameters using this property encourages task specific decoders to optimize different parts of the feature extractor, thus reducing competition. We evaluate our method with classification and regression tasks on the multiDigitMNIST, NYUv2 and SUN RGB-D datasets where we obtain competitive results.

show abstract

“…We note that the above penalty can also be thought of as a network regularization. Similar gradient penalties are used in machine learning to improve generalization ability and to improve the robustness to adversarial attacks [36]. The use of gradient penalty is observed to be qualitatively equivalent to penalizing the norm of the weights of the network.…”

Section: B Distance/network Regularizationmentioning

confidence: 95%

Dynamic imaging using a deep generative SToRM (Gen-SToRM) model

Zou

Ahmed

Nagpal

et al. 2021

Preprint

View full text Add to dashboard Cite

We introduce a generative smoothness regularization on manifolds (SToRM) model for the recovery of dynamic image data from highly undersampled measurements. The model assumes that the images in the dataset are non-linear mappings of low-dimensional latent vectors. We use the deep convolutional neural network (CNN) to represent the non-linear transformation. The parameters of the generator as well as the lowdimensional latent vectors are jointly estimated only from the undersampled measurements. This approach is different from traditional CNN approaches that require extensive fully sampled training data. We penalize the norm of the gradients of the nonlinear mapping to constrain the manifold to be smooth, while temporal gradients of the latent vectors are penalized to obtain a smoothly varying time-series. The proposed scheme brings in the spatial regularization provided by the convolutional network. The main benefit of the proposed scheme is the improvement in image quality and the orders-of-magnitude reduction in memory demand compared to traditional manifold models. To minimize the computational complexity of the algorithm, we introduce an efficient progressive training-in-time approach and an approximate cost function. These approaches speed up the image reconstructions and offers better reconstruction performance.

show abstract

Gradient Regularization Improves Accuracy of Discriminative Models

Cited by 14 publications

References 9 publications

Robust Learning with Jacobian Regularization

Robust Learning with Jacobian Regularization

Regularizing Deep Multi-Task Networks using Orthogonal Gradients

Dynamic imaging using a deep generative SToRM (Gen-SToRM) model

Contact Info

Product

Resources

About