On the Inherent Regularization Effects of Noise Injection During Training

Abstract

Randomly perturbing networks during the training process is a commonly used approach to improving generalization performance. In this paper, we present a theoretical study of one particular way of random perturbation, which corresponds to injecting artificial noise to the training data. We provide a precise asymptotic characterization of the training and generalization errors of such randomly perturbed learning problems on a random feature model. Our analysis shows that Gaussian noise injection in the training process is equivalent to introducing a weighted ridge regularization, when the number of noise injections tends to infinity. The explicit form of the regularization is also given. Numerical results corroborate our asymptotic predictions, showing that they are accurate even in moderate problem dimensions. Our theoretical predictions are based on a new correlated Gaussian equivalence conjecture that generalizes recent results in the study of random feature models.

On Gradient Descent Training Under Data Augmentation with On-Line Noisy Copies

“…We also need to analyze the effect of DA with noisy copies on the training process of neural networks. The analysis on the regularization effect of layered and convolutional neural networks may be looked at in [4]. On the other hand, a mixture of apparent acceleration and regularization effect obtained by DA may have a different effect on the training process of neural networks.…”

“…DA with on-line noisy copies can be viewed as a kind of noise injection into the training process, in which there are methods of injecting noise into inputs, weights and hidden output; e.g. [2], [4], [9], [15], [16]. In this direction, there are considerable works that discuss how to overcome adversarial examples particularly in images; e.g., see [16].…”

“…In this direction, there are considerable works that discuss how to overcome adversarial examples particularly in images; e.g., see [16]. Among these, [4] is the most relevant work that provides an asymptotic characterization of the training and generalization errors of input noise injection under a random feature model. This is more general than our result as it considers a nonlinear model but it does not consider the DA setting and a difference in training criterion.…”

“…Fig 4. Training curves for a naive SGD (referred to as SGD) and SGD with off-line DA (referred to as SGD-DA) under different datasets and batch sizes (BS).…”