Noise injection for training artificial neural networks: A comparison with weight decay and early stopping

Abstract

The purpose of this study was to investigate the effect of a noise injection method on the "overfitting" problem of artificial neural networks (ANNs) in two-class classification tasks. The authors compared ANNs trained with noise injection to ANNs trained with two other methods for avoiding overfitting: weight decay and early stopping. They also evaluated an automatic algorithm for selecting the magnitude of the noise injection. They performed simulation studies of an exclusive-or classification task with training datasets of 50, 100, and 200 cases (half normal and half abnormal) and an independent testing dataset of 2000 cases. They also compared the methods using a breast ultrasound dataset of 1126 cases. For simulated training datasets of 50 cases, the area under the receiver operating characteristic curve (AUC) was greater (by 0.03) when training with noise injection than when training without any regularization, and the improvement was greater than those from weight decay and early stopping (both of 0.02). For training datasets of 100 cases, noise injection and weight decay yielded similar increases in the AUC (0.02), whereas early stopping produced a smaller increase (0.01). For training datasets of 200 cases, the increases in the AUC were negligibly small for all methods (0.005). For the ultrasound dataset, noise injection had a greater average AUC than ANNs trained without regularization and a slightly greater average AUC than ANNs trained with weight decay. These results indicate that training ANNs with noise injection can reduce overfitting to a greater degree than early stopping and to a similar degree as weight decay.

Effect of Additive Noise for Multi-Layered Perceptron with AutoEncoders

“…Random Noise injection to the weights or the hidden units has been utilized in many neural network researches for many years [15], [19]- [21]. Kurita et al [9] injected noise into the hidden layers of an MLP and showed the network ability to get automatically structurized by simply adding the noises and therefore improving the generalization ability of the network.…”

Severity Prediction of Traffic Accidents with Recurrent Neural Networks

“…Therefore, to avoid over-fitting in the proposed RNN model, three standard techniques were used. These were Gaussian noise injection into the training data [32], using a ReLU activation function in the hidden layers [20], and subsequently applying the dropout technique [29]. Dropout leads to big improvements in the prediction performance of the model.…”

Combined inverse-forward artificial neural networks for fast and accurate estimation of the diffusion coefficients of cartilage based on multi-physics models

“…The first element of the inverse-forward ANNs is the inverse ANN which is trained with noisy data and is responsible to filter the noise, while the second element is the forward ANN which eliminates the necessity of FEM. Training with noise-free FEM data of the inverse ANN would result in overfitting and its subsequent failure (Arbabi et al, 2015a;Zur et al, 2009). The inverse ANN trained with noisy data (Gaussian random noise) is most sensitive to the general trend of the experimental data without being influenced by small deviations from FEM caused by uncertainties involved in the experimental data.…”

“…Our previous study showed that ANNs are very sensitive to any deviations from the underlying computational model that is used for their training (Arbabi et al, 2015a). To alleviate this problem, the training data of ANNs can be contaminated with some level of noise (Arbabi et al, 2015a;Derks et al, 1997;Zur et al, 2009) to increase the robustness of ANN. Therefore, we trained the ANN using the input concentration vs. time curves contaminated with different levels of Gaussian noise, i.e.…”