ReLU Deep Neural Networks and Linear Finite Elements

He, Juncai; Li, Lin; Xu, Jinchao; Zheng, Chunyue

doi:10.48550/arxiv.1807.03973

Cited by 62 publications

(93 citation statements)

References 33 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…During training, multilayer perceptrons (MLPs) construct data-driven bases with no reference to underlying geometry [Cyr et al, 2020, He et al, 2018. It has been proven that for MLPs of increasing width and depth, weights and biases exist for which the Sobolev norm of approximation error converge algebraically.…”

Section: Introductionmentioning

confidence: 99%

Probabilistic partition of unity networks: clustering based deep approximation

Trask,

Gulian,

Huang

et al. 2021

Preprint

View full text Add to dashboard Cite

Partition of unity networks (POU-Nets) have been shown capable of realizing algebraic convergence rates for regression and solution of PDEs, but require empirical tuning of training parameters. We enrich POU-Nets with a Gaussian noise model to obtain a probabilistic generalization amenable to gradient-based minimization of a maximum likelihood loss. The resulting architecture provides spatial representations of both noiseless and noisy data as Gaussian mixtures with closed form expressions for variance which provides an estimator of local error. The training process yields remarkably sharp partitions of input space based upon correlation of function values. This classification of training points is amenable to a hierarchical refinement strategy that significantly improves the localization of the regression, allowing for higher-order polynomial approximation to be utilized. The framework scales more favorably to large data sets as compared to Gaussian process regression and allows for spatially varying uncertainty, leveraging the expressive power of deep neural networks while bypassing expensive training associated with other probabilistic deep learning methods. Compared to standard deep neural networks, the framework demonstrates hp-convergence without the use of regularizers to tune the localization of partitions. We provide benchmarks quantifying performance in high/low-dimensions, demonstrating that convergence rates depend only on the latent dimension of data within high-dimensional space. Finally, we introduce a new open-source data set of PDE-based simulations of a semiconductor device and perform unsupervised extraction of a physically interpretable reduced-order basis.

show abstract

Section: Introductionmentioning

confidence: 99%

Probabilistic partition of unity networks: clustering based deep approximation

Trask,

Gulian,

Huang

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…The universal approximation theorem ( [17,33]) clarifies that every continuous function on a compact domain can be uniformly approximated by shallow neural networks with continuous, non-polynomial activation functions. The relationship between ReLU-DNN and linear finite element function was studied in [29]. More results have been established in [4,50,60,11,66] for activation functions with a certain regularity, and these approximation errors were given in the sense of L p norm.…”

Section: Neural Networkmentioning

confidence: 99%

“…DNNs produce a large class of nonlinear functions through compositional construction. Due to their powerful universal approximation ability, in recent years, DNNs have been applied for solving partial differential equations (PDEs), and several DNNbased methods ( [21,5,28,29,22,56,16,23,37,45,42,44]) were proposed to overcome the difficulty so-called the "curse of dimensionality" of the traditional PDE solvers such as finite element method (FEM), which requires a discretization of the interested domain, while the number of the mesh points will increase exponentially fast with respect to the problem dimension and make it quickly become computationally intractable. In such a situation, the generation of meshes is very time-consuming as well.…”

Section: Introductionmentioning

confidence: 99%

Deep Petrov-Galerkin Method for Solving Partial Differential Equations

Shang¹,

Wang²,

Sun³

2022

Preprint

View full text Add to dashboard Cite

Deep neural networks are powerful tools for approximating functions, and they are applied to successfully solve various problems in many fields. In this paper, we propose a neural network-based numerical method to solve partial differential equations. In this new framework, the method is designed on weak formulations, and the unknown functions are approximated by deep neural networks and test functions can be chosen by different approaches, for instance, basis functions of finite element methods, neural networks, and so on. Because the spaces of trial function and test function are different, we name this new approach by Deep Petrov-Galerkin Method (DPGM). The resulted linear system is not necessarily to be symmetric and square, so the discretized problem is solved by a least-square method. Take the Poisson problem as an example, mixed DPGMs based on several mixed formulations are proposed and studied as well.In addition, we apply the DPGM to solve two classical time-dependent problems based on the space-time approach, that is, the unknown function is approximated by a neural network, in which temporal variable and spatial variables are treated equally, and the initial conditions are regarded as boundary conditions for the space-time domain. Finally, several numerical examples are presented to show the performance of the DPGMs, and we observe that this new method outperforms traditional numerical methods in several aspects: compared to the finite element method and finite difference method, DPGM is much more accurate with respect to degrees of freedom; this method is mesh-free, and can be implemented easily; mixed DPGM has good flexibility to handle different boundary conditions; DPGM can solve the time-dependent problems by the space-time approach naturally and efficiently. The proposed deep Petrov-Galerkin method shows strong potential in the field of numerical methods for partial differential equations.

show abstract

“…For instance, in finite-deformation simulations using finite elements, the optimal nodal locations and the solution coefficients have both been simultaneously treated as unknowns in the minimization of the potential energy functional [38]. Since the PINN approximation that is composed by the ReLU activation function can exactly represent piecewise affine functions (Delaunay basis functions) [39], one can view the ReLU network solution as a variational r-adaptive finite element solution procedure. Instead of refining elements in h-adaptive finite elements, adaptive solutions can be realized via a basis refinement strategy that has advantages (for example, 'hanging nodes' are a nonissue), which was put forth by Grinspun [40], and a similar basis refinement perspective can be associated with a multilayer neural network [41].…”

Section: Introductionmentioning

confidence: 99%

Exact imposition of boundary conditions with distance functions in physics-informed deep neural networks

Sukumar,

Srivastava

2021

Preprint

View full text Add to dashboard Cite

In this paper, we introduce a new approach based on distance fields to exactly impose boundary conditions in physicsinformed deep neural networks. The challenges in satisfying Dirichlet boundary conditions in meshfree and particle methods are well-known. This issue is also pertinent in the development of physics informed neural networks (PINN) for the solution of partial differential equations. We introduce geometry-aware trial functions in artifical neural networks to improve the training in deep learning for partial differential equations. To this end, we use concepts from constructive solid geometry (R-functions) and generalized barycentric coordinates (mean value potential fields) to construct φ(x), an approximate distance function to the boundary of a domain in R d . To exactly impose homogeneous Dirichlet boundary conditions, the trial function is taken as φ(x) multiplied by the PINN approximation, and its generalization via transfinite interpolation is used to a priori satisfy inhomogeneous Dirichlet (essential), Neumann (natural), and Robin boundary conditions on complex geometries. In doing so, we eliminate modeling error associated with the satisfaction of boundary conditions in a collocation method and ensure that kinematic admissibility is met pointwise in a Ritz method. With this new ansatz, the training for the neural network is simplified: sole contribution to the loss function is from the residual error at interior collocation points where the governing equation is required to be satisfied. Numerical solutions are computed using strong form collocation and Ritz minimization. To convey the main ideas and to assess the accuracy of the approach, we present numerical solutions for linear and nonlinear boundary-value problems over convex and nonconvex polygonal domains as well as over domains with curved boundaries. Benchmark problems in one dimension for linear elasticity, advection-diffusion, and beam bending; and in two dimensions for the steady-state heat equation, Laplace equation, biharmonic equation (Kirchhoff plate bending), and the nonlinear Eikonal equation are considered. The construction of approximate distance functions using R-functions extends to higher dimensions, and we showcase its use by solving a Poisson problem with homogeneneous Dirichlet boundary conditions over the four-dimensional hypercube. The proposed approach consistently outperforms a standard PINN-based collocation method, which underscores the importance of exactly (a priori) satisfying the boundary condition when constructing a loss function in PINN. This study provides a pathway for meshfree analysis to be conducted on the exact geometry without domain discretization.

show abstract

ReLU Deep Neural Networks and Linear Finite Elements

Cited by 62 publications

References 33 publications

Probabilistic partition of unity networks: clustering based deep approximation

Probabilistic partition of unity networks: clustering based deep approximation

Deep Petrov-Galerkin Method for Solving Partial Differential Equations

Exact imposition of boundary conditions with distance functions in physics-informed deep neural networks

Contact Info

Product

Resources

About