Enhancing Function Approximation Abilities of Neural Networks by Training Derivatives

Abstract

A method to increase the precision of feedforward networks is proposed. It requires a prior knowledge of a target function derivatives of several orders and uses this information in gradient based training. Forward pass calculates not only the values of the output layer of a network but also their derivatives. The deviations of those derivatives from the target ones are used in an extended cost function and then backward pass calculates the gradient of the extended cost with respect to weights, which can then be used by any weights update algorithm. Despite a substantial increase in arithmetic operations per pattern (if compared to the conventional training), the extended cost allows to obtain 140-1000 times more accurate approximation for simple cases if the total number of operations is equal. This precision also happens to be out of reach for the regular cost function. The method fits well into the procedure of solving differential equations with neural networks. Unlike training a network to match some target mapping, which requires an explicit use of the target derivatives in the extended cost function, the cost function for solving a differential equation is based on the deviation of the equation's residual from zero and thus can be extended by differentiating the equation itself, which does not require any prior knowledge. Solving an equation with such a cost resulted in 13 times more accurate result and could be done with 3 times larger grid step. GPU-efficient algorithm for calculating the gradient of the extended cost function is proposed.

Neural networks catching up with finite differences in solving partial differential equations in higher dimensions

“…The input of such network is considered as a vector of independent variables and the output as value of solution. All necessary derivatives of output with respect to input and of cost function with respect to weights can be calculated by the extended backpropagation procedure [2]. Including boundary conditions into cost function itself usually does not produce very accurate results [24], so a process of function substitution is required [18].…”

“…They are propagated backward in order to obtain gradient of E with respect to the weights of each layer. The whole procedure is described in [2]. One can mention a few features of the neural network approach:…”

Analytical solution of stochastic differential equation by multilayer perceptron neural network approximation of Fokker–Planck equation

“…Also, the differences between the mean and the variance calculated by Equations (27) and (28) and the MC simulations are shown in Figure 2a,b.…”

An energy approach to the solution of partial differential equations in computational mechanics via machine learning: Concepts, implementation and applications

“…whereû andv are obtained from the neural network. The solution of u and v are chosen in such a way that it satisfies all the boundary conditions as in [24,25]. As a consequence, no component corresponding to the boundary loss is needed in the loss function.…”

“…whereφ is given by the neural network. The solution of φ is chosen in such a way that it satisfies all the boundary conditions as in [24,25]. This ensures that the boundary conditions are satisfied during the training of the network.…”