Preventing Overfitting by Training Derivatives

Abstract

Derivative training is a well-known method to improve the accuracy of neural networks. In the forward pass, not only the output values are computed, but also their derivatives, and their deviations from the target derivatives are included in the cost function, which is minimized with respect to the weights by a gradient-based algorithm. So far, this method has been implemented for relatively low-dimensional tasks. In this study, we apply the approach to the problem of image analysis. We consider the task of reconstructing the vertices of a cube based on its image. By training the derivatives with respect to the 6 degrees of freedom of the cube, we obtain 25 times more accurate results for noiseless inputs. The derivatives also provide important insights into the robustness problem, which is currently understood in terms of two types of network vulnerabilities. The first type is small perturbations that dramatically change the output, and the second type is substantial image changes that the network erroneously ignores. They are currently considered as conflicting goals, since conventional training methods produce a trade-off. The first type can be analyzed via the gradient of the network, but the second type requires human evaluation of the inputs, which is an oracle substitute. For the task at hand, the nearest neighbor oracle can be defined, and the knowledge of derivatives allows it to be expanded into Taylor series. This allows to perform the first-order robustness analysis that unifies both types of vulnerabilities, and to implement robust training that eliminates any trade-offs, so that accuracy and robustness are limited only by network capacity.

Synthesis optimization and adsorption modeling of biochar for pollutant removal via machine learning

“…Over-fitting and underfitting are common fitting problems for model training (Dahan and Keller 2021;Graham et al 2020;Yuan et al 2020). Over-fitting takes the local features of training data set as the whole features, and the reason is that too many parameters of model make the training error small and the test error large (Yildirim and Ozkale 2021;Avrutskiy 2020). When under-fitting occurs, the learning ability of model is weak and the basic features of training data set cannot be learned (Handelman et al 2019;Ahmed and Isa 2017;Van Calster and Vickers 2015).…”

Improve Generalization in Deep Neural Network using Multivariate Statistical Knowledge Transformation

“…The capacity of a model is defined by its ability to fit various functions. The model architecture, number of hidden layers, weight size, and hypothesis space size affect the capacity of the model [10,39,40]. In addition, weight decay is a term added to the objective function to penalize the high-order weights [10,27], In the early stopping technique, training should be stopped before weights become large to avoid hidden units being in their non-linear ranges, resulting in high capacity [29].…”

“…Data augmentation is an effective approach for magnifying small datasets by inverting, trimming, rotating, zooming, and adjusting brightness, sharpness, and contrast. These approaches rely on existing samples to create new samples from the entire training dataset [10,35].…”

Analytical solution of stochastic differential equation by multilayer perceptron neural network approximation of Fokker–Planck equation

“…The first idea of approximation of a differential equation by MLP‐NN was proposed by IE Lagaris with one hidden‐layer network . Later Avrutskiy develops the idea by using more than one hidden layer MLP‐NN . The basic idea of this paper is inspired by these works.…”

“…In , the single hidden layer network is applied to find the solution of differential equations. Recently, more than one‐hidden‐layer structure network is also investigated for both enhancing the function approximation power of the neural network as well as finding the solution of PDEs . In a special form of the solution for differential equations was chosen to satisfy the initial and boundary conditions.…”