Numerical Approximation of Partial Differential Equations by a Variable Projection Method with Artificial Neural Networks

Abstract

We present a method for solving linear and nonlinear partial differential equations (PDE) based on the variable projection framework and artificial neural networks. For linear PDEs, enforcing the boundary/initial value problem on the collocation points gives rise to a separable nonlinear least squares problem about the network coefficients. We reformulate this problem by the variable projection approach to eliminate the linear output-layer coefficients, leading to a reduced problem about the hidden-layer coefficients only. The reduced problem is solved first by the nonlinear least squares method to determine the hidden-layer coefficients, and then the output-layer coefficients are computed by the linear least squares method. For nonlinear PDEs, enforcing the boundary/initial value problem on the collocation points gives rise to a nonlinear least squares problem that is not separable, which precludes the variable projection strategy for such problems. To enable the variable projection approach for nonlinear PDEs, we first linearize the problem with a Newton iteration, using a particular linearization formulated in terms of the updated approximation field. The linearized system is solved by the variable projection framework together with artificial neural networks. Upon convergence of the Newton iteration, the neural-network coefficients provide the representation of the solution field to the original nonlinear problem. We present ample numerical examples with linear and nonlinear PDEs to demonstrate the performance of the method developed herein. For smooth field solutions, the errors of the current method decrease exponentially as the number of collocation points or the number of output-layer coefficients increases. We compare extensively the current method with the extreme learning machine (ELM) method from a previous work. Under identical conditions and network configurations, the current method exhibits an accuracy significantly superior to the ELM method.

Numerical Computation of Partial Differential Equations by Hidden-Layer Concatenated Extreme Learning Machine

“…v(ϑ, α, x) = u(θ, β, x). We thus conclude that u(θ, β, x) ∈ U (Ω, M 2 , σ, ϑ) and the relation (13) holds.…”

“…The authors observe that the ELM method exhibits a better accuracy than the finite difference and the finite element method. Another recent development related to ELM is [13], in which a method based on the variable projection strategy is proposed for solving linear and nonlinear PDEs with artificial neural networks. For linear PDEs, the neural-network representation of the PDE solution leads to a separable nonlinear least squares problem, which is then reformulated to eliminate the output-layer coefficients, leading to a reduced problem about the hidden-layer coefficients only.…”

“…For linear PDEs, the neural-network representation of the PDE solution leads to a separable nonlinear least squares problem, which is then reformulated to eliminate the output-layer coefficients, leading to a reduced problem about the hidden-layer coefficients only. The reduced problem is solved first by the nonlinear least squares method to determine the hidden-layer coefficients, and the output-layer coefficients are then computed by the linear least squares method [13]. For nonlinear PDEs, the problem is first linearized by the Newton's method with a particular linearization form, and the linearized system is solved by the variable projection framework together with neural networks.…”

“…The ELM method can be considered as a special case of the variable projection, i.e. with zero iteration when solving the reduced problem for the hidden-layer coefficients [13]. It is shown in [13] that the variable projection method exhibits an accuracy significantly superior to the ELM under identical conditions and network configurations.…”

A Method for Computing Inverse Parametric Pde Problems with Random-Weight Neural Networks

“…This subsection outlines two algorithms for computing (α, β), both based on the variable projection (VarPro) idea [23,24,18] but with different formulations. In the first formulation (VarPro-F1), the inverse parameters (α) are eliminated from the problem to attain a reduced problem about β only.…”