Optimization free neural network approach for solving ordinary and partial differential equations

Panghal, Shagun; Kumar, Manoj

doi:10.1007/s00366-020-00985-1

Cited by 46 publications

(12 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The use of ELM for solving linear partial differential equations has been discussed in a number of previous works; see e.g. [31,10,6] and the references therein. For the sake of completeness, we summarize the main procedure below, and we refer the reader to e.g.…”

Section: Solving Linear Differential Equations With Combined Elm/modbipmentioning

confidence: 99%

A Modified Batch Intrinsic Plasticity Method for Pre-training the Random Coefficients of Extreme Learning Machines

Dong,

2021

Preprint

View full text Add to dashboard Cite

In extreme learning machines (ELM) the hidden-layer coefficients are randomly set and fixed, while the output-layer coefficients of the neural network are computed by a least squares method. The randomlyassigned coefficients in ELM are known to influence its performance and accuracy significantly. In this paper we present a modified batch intrinsic plasticity (modBIP) method for pre-training the random coefficients in the ELM neural networks. The current method is devised based on the same principle as the batch intrinsic plasticity (BIP) method, namely, by enhancing the information transmission in every node of the neural network. It differs from BIP in two prominent aspects. First, modBIP does not involve the activation function in its algorithm, and it can be applied with any activation function in the neural network. In contrast, BIP employs the inverse of the activation function in its construction, and requires the activation function to be invertible (or monotonic). The modBIP method can work with the oftenused non-monotonic activation functions (e.g. Gaussian, swish, Gaussian error linear unit, and radial-basis type functions), with which BIP breaks down. Second, modBIP generates target samples on random intervals with a minimum size, which leads to highly accurate computation results when combined with ELM. The combined ELM/modBIP method is markedly more accurate than ELM/BIP in numerical simulations. Ample numerical experiments are presented with shallow and deep neural networks for function approximation and boundary/initial value problems with partial differential equations. They demonstrate that the combined ELM/modBIP method produces highly accurate simulation results, and that its accuracy is insensitive to the random-coefficient initializations in the neural network. This is in sharp contrast with the ELM results without pre-training of the random coefficients.

show abstract

Section: Solving Linear Differential Equations With Combined Elm/modbipmentioning

confidence: 99%

A Modified Batch Intrinsic Plasticity Method for Pre-training the Random Coefficients of Extreme Learning Machines

Dong,

2021

Preprint

View full text Add to dashboard Cite

show abstract

“…[33,20,30,21,47]), which can be traced to Turing's unorganized machine and Rosenblatt's perceptron [42,35] and have witnessed a revival in neuro-computations in recent years. The application of ELM to function approximations and linear differential equations have been considered in several recent works [1,45,40,32,29,10]. Domain decomposition has found widespread applications in classical numerical methods [39,41,4,8,5].…”

Section: Introductionmentioning

confidence: 99%

Local Extreme Learning Machines and Domain Decomposition for Solving Linear and Nonlinear Partial Differential Equations

Dong,

2020

Preprint

140

View full text Add to dashboard Cite

We present a neural network-based method for solving linear and nonlinear partial differential equations, by combining the ideas of extreme learning machines (ELM), domain decomposition and local neural networks. The field solution on each sub-domain is represented by a local feed-forward neural network, and C k continuity with an appropriate integer k is imposed on the sub-domain boundaries. Each local neural network consists of a small number (one or more) of hidden layers, while its last hidden layer can be wide. The weight/bias coefficients in all the hidden layers of the local neural networks are pre-set to random values and fixed throughout the computation, and only the weight coefficients in the output layers of the local neural networks are adjustable training parameters. The overall neural network is trained by a linear or nonlinear least squares computation, not by the back-propagation type algorithms. We introduce a block time-marching scheme together with the presented method for long-time simulations of time-dependent linear/nonlinear partial differential equations. The current method exhibits a clear sense of convergence with respect to the degrees of freedom in the neural network. Its numerical errors typically decrease exponentially or nearly exponentially as the number of degrees of freedom (e.g. the number of training parameters, number of training data points, number of sub-domains) in the system increases. Extensive numerical experiments have been performed to demonstrate the computational performance of the current method and to study the effects of the simulation parameters. We also present results to demonstrate its capability for long-time dynamic simulations with certain test problems. We compare the presented method with the deep Galerkin method (DGM) and the physics-informed neural network (PINN) method in terms of the accuracy and computational cost. The current method exhibits a clear superiority, with its numerical errors and network training time considerably smaller (typically by orders of magnitude) than those of DGM and PINN. We also compare the current method with the classical finite element method (FEM). The computational performance of the current method is on par with, and often exceeds, the FEM performance in terms of the accuracy and computational cost. To achieve the same accuracy, the network training time of the current method is comparable to, and oftentimes less than, the FEM computation time. Under the same computational cost (training/computation time), the numerical errors of the current method are comparable to, and oftentimes markedly smaller than, the FEM errors.

show abstract

“…In ELM one assigns random values to and fixes the hidden-layer coefficients, and only allows the output-layer (assumed to be linear) coefficients to be trainable. For linear problems, the resultant system becomes linear with respect to the output-layer coefficients, which can then be determined by a linear least squares method [26,41,14,10,11,4]. Randomweight neural networks similarly possess a universal approximation property.…”

mentioning

confidence: 99%

On Computing the Hyperparameter of Extreme Learning Machines: Algorithm and Application to Computational PDEs, and Comparison with Classical and High-Order Finite Elements

Dong,

Yang

2021

Preprint

View full text Add to dashboard Cite

We consider the use of extreme learning machines (ELM) for computational partial differential equations (PDE). In ELM the hidden-layer coefficients in the neural network are assigned to random values generated on [−Rm, Rm] and fixed, where Rm is a user-provided constant, and the output-layer coefficients are trained by a linear or nonlinear least squares computation. We present a method for computing the optimal or near-optimal value of Rm based on the differential evolution algorithm. This method seeks the optimal Rm by minimizing the residual norm of the linear or nonlinear algebraic system that results from the ELM representation of the PDE solution and that corresponds to the ELM least squares solution to the system. It amounts to a pre-processing procedure that determines a near-optimal Rm, which can be used in ELM for solving linear or nonlinear PDEs. The presented method enables us to illuminate the characteristics of the optimal Rm for two types of ELM configurations: (i) Single-Rm-ELM, corresponding to the conventional ELM method in which a single Rm is used for generating the random coefficients in all the hidden layers, and (ii) Multi-Rm-ELM, corresponding to a modified ELM method in which multiple Rm constants are involved with each used for generating the random coefficients of a different hidden layer. We adopt the optimal Rm from this method and also incorporate other improvements into the ELM implementation. In particular, here we compute all the differential operators involving the output fields of the last hidden layer by a forward-mode auto-differentiation, as opposed to the reverse-mode auto-differentiation in a previous work. These improvements significantly reduce the network training time and enhance the ELM performance. We systematically compare the computational performance of the current improved ELM with that of the finite element method (FEM), both the classical second-order FEM and the high-order FEM with Lagrange elements of higher degrees, for solving a number of linear and nonlinear PDEs. It is shown that the current improved ELM far outperforms the classical FEM. Its computational performance is comparable to that of the high-order FEM for smaller problem sizes, and for larger problem sizes the ELM markedly outperforms the high-order FEM.

show abstract

Optimization free neural network approach for solving ordinary and partial differential equations

Cited by 46 publications

References 20 publications

A Modified Batch Intrinsic Plasticity Method for Pre-training the Random Coefficients of Extreme Learning Machines

A Modified Batch Intrinsic Plasticity Method for Pre-training the Random Coefficients of Extreme Learning Machines

Local Extreme Learning Machines and Domain Decomposition for Solving Linear and Nonlinear Partial Differential Equations

On Computing the Hyperparameter of Extreme Learning Machines: Algorithm and Application to Computational PDEs, and Comparison with Classical and High-Order Finite Elements

Contact Info

Product

Resources

About