Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism

McClenny, Levi D.; Braga-Neto, Ulisses

doi:10.48550/arxiv.2009.04544

Cited by 67 publications

(93 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…(2). The interested reader is directed to [98][99][100] for online and offline adaptive weight techniques. Following determination of θ, u θ and λ θ can be evaluated at any x in Ω, Ω λ , respectively.…”

Section: Solving Forward and Mixed Pde Problems: Overview Of Pinn Met...mentioning

confidence: 99%

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Psaros¹,

Meng²,

Zou³

et al. 2022

Preprint

View full text Add to dashboard Cite

Neural networks (NNs) are currently changing the computational paradigm on how to combine data with mathematical laws in physics and engineering in a profound way, tackling challenging inverse and ill-posed problems not solvable with traditional methods. However, quantifying errors and uncertainties in NN-based inference is more complicated than in traditional methods. This is because in addition to aleatoric uncertainty associated with noisy data, there is also uncertainty due to limited data, but also due to NN hyperparameters, overparametrization, optimization and sampling errors as well as model misspecification. Although there are some recent works on uncertainty quantification (UQ) in NNs, there is no systematic investigation of suitable methods towards quantifying the total uncertainty effectively and efficiently even for function approximation, and there is even less work on solving partial differential equations and learning operator mappings between infinite-dimensional function spaces using NNs. In this work, we present a comprehensive framework that includes uncertainty modeling, new and existing solution methods, as well as evaluation metrics and post-hoc improvement approaches. To demonstrate the applicability and reliability of our framework, we present an extensive comparative study in which various methods are tested on prototype problems, including problems with mixed input-output data, and stochastic problems in high dimensions. In the Appendix, we include a comprehensive description of all the UQ methods employed, which we will make available as open-source library of all codes included in this framework.

show abstract

Section: Solving Forward and Mixed Pde Problems: Overview Of Pinn Met...mentioning

confidence: 99%

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Psaros¹,

Meng²,

Zou³

et al. 2022

Preprint

View full text Add to dashboard Cite

show abstract

“…Properly setting the weight vector λ = (λ r,0 , λ r,1 , λ ic , λ bc ) is critical to enhance the trainability of constrained neural networks [22,23], but searching the optimal weight vector through manual tuning is infeasible. We adapt the uncertainty weighing method in [24] to solve the problem.…”

Section: Lower Bound Constrained Uncertainty Weightingmentioning

confidence: 99%

Solving Partial Differential Equations with Point Source Based on Physics-Informed Neural Networks

Huang¹,

Liu²,

Shi³

et al. 2021

Preprint

View full text Add to dashboard Cite

In recent years, deep learning technology has been used to solve partial differential equations (PDEs), among which the physics-informed neural networks (PINNs) emerges to be a promising method for solving both forward and inverse PDE problems. PDEs with a point source that is expressed as a Dirac delta function in the governing equations are mathematical models of many physical processes. However, they cannot be solved directly by conventional PINNs method due to the singularity brought by the Dirac delta function. We propose a universal solution to tackle this problem with three novel techniques. Firstly the Dirac delta function is modeled as a continuous probability density function to eliminate the singularity; secondly a lower bound constrained uncertainty weighting algorithm is proposed to balance the PINNs losses between point source area and other areas; and thirdly a multi-scale deep neural network with periodic activation function is used to improve the accuracy and convergence speed of the PINNs method. We evaluate the proposed method with three representative PDEs, and the experimental results show that our method outperforms existing deep learning-based methods with respect to the accuracy, the efficiency and the versatility.

show abstract

“…Others have sought to extend PINNs to be applicable to a wider variety of PDE problems, including PDEs with fractional powers [10] or where solutions to parameterized PDEs are desired [11]. At the same time, efforts have been made to improve convergence rates with adaptive activation functions [12,13,14], intelligent selection of loss weightings [15], or subdomain approaches that use multiple PINNs across the domain of interest stitched together with appropriate interface conditions [16,17]. Related efforts have also attempted to quantify the generalization error of these methods for specific problems [4,18,19,20] and explain the convergence difficulties that PINNs face when solving some PDEs [15].…”

Section: Related Workmentioning

confidence: 99%

Learning To Estimate Regions Of Attraction Of Autonomous Dynamical Systems Using Physics-Informed Neural Networks

Scharzenberger¹,

Hays²

2021

Preprint

View full text Add to dashboard Cite

When learning to perform motor tasks in a simulated environment, neural networks must be allowed to explore their action space to discover new potentially viable solutions. However, in an online learning scenario with physical hardware, this exploration must be constrained by relevant safety considerations in order to avoid damage to the agent's hardware and environment. We aim to address this problem by training a neural network, which we will refer to as a "safety network", to estimate the region of attraction (ROA) of a controlled autonomous dynamical system. This safety network can thereby be used to quantify the relative safety of proposed control actions and prevent the selection of damaging actions. Here we present our development of the safety network by training an artificial neural network (ANN) to represent the ROA of several autonomous dynamical system benchmark problems. The training of this network is predicated upon both Lyapunov theory and neural solutions to partial differential equations (PDEs). By learning to approximate the viscosity solution to a specially chosen PDE that contains the dynamics of the system of interest, the safety network learns to approximate a particular function, similar to a Lyapunov function, whose zero level set is the boundary of the ROA. We train our safety network to solve these PDEs in a semi-supervised manner following a modified version of the Physics Informed Neural Network (PINN) approach, utilizing a loss function that penalizes disagreement with the PDE's initial and boundary conditions, as well as non-zero residual and variational terms. In future work we intend to apply this technique to reinforcement learning agents during motor learning tasks.

show abstract

Self-Adaptive Physics-Informed Neural Networks using a Soft Attention Mechanism

Cited by 67 publications

References 13 publications

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Uncertainty Quantification in Scientific Machine Learning: Methods, Metrics, and Comparisons

Solving Partial Differential Equations with Point Source Based on Physics-Informed Neural Networks

Learning To Estimate Regions Of Attraction Of Autonomous Dynamical Systems Using Physics-Informed Neural Networks

Contact Info

Product

Resources

About