Inverse Dirichlet weighting enables reliable training of physics informed neural networks

Abstract

We characterize and remedy a failure mode that may arise from multi-scale dynamics with scale imbalances during training of deep neural networks, such as Physics Informed Neural Networks (PINNs). PINNs are popular machine-learning templates that allow for seamless integration of physical equation models with data. Their training amounts to solving an optimization problem over a weighted sum of data-fidelity and equation-fidelity objectives. Conflicts between objectives can arise from scale imbalances, heteroscedasticity in the data, stiffness of the physical equation, or from catastrophic interference during sequential training. We explain the training pathology arising from this and propose a simple yet effective inverse Dirichlet weighting strategy to alleviate the issue. We compare with Sobolev training of neural networks, providing the baseline of analytically ε-optimal training. We demonstrate the effectiveness of inverse Dirichlet weighting in various applications, including a multi-scale model of active turbulence, where we show orders of magnitude improvement in accuracy and convergence over conventional PINN training. For inverse modeling using sequential training, we find that inverse Dirichlet weighting protects a PINN against catastrophic forgetting.

Investigating molecular transport in the human brain from MRI with physics-informed neural networks

“…Our results are in line with the frequent observation that the PDE loss weight is an important hyperparameter. Several works have put forth methodologies to choose the weights adaptively during training 8 – 11 , but in practice they have also been chosen via trial-and-error 43 , 55 , 56 . However, in settings with noisy data, it can not be expected that both data loss and PDE loss become zero.…”

“…Another challenge in training PINNs is balancing boundary, initial and PDE loss terms. This challenge has been addressed by adaptive weighting strategies 8 – 11 , as well as theory of functional connections 12 , 13 . Despite these challenges, the effectiveness of the method has been demonstrated in a wide range of works, examples include turbulent flows 14 , heat transfer 15 , epidemiological compartmental models 16 or stiff chemical systems 17 .…”

PINN Training using Biobjective Optimization: The Trade-off between Data Loss and Residual Loss

“…Maddu et al [13] suggest a multiobjective descent method that adaptively updates the weights using an inverse Dirichlet strategy to avoid premature termination. While they do not discuss convergence guarantees, their numerical results show a good performance in comparison with recent adaptations of multiobjective descent methods [14,15] to PINN training [16].…”

On the Role of Fixed Points of Dynamical Systems in Training Physics-Informed Neural Networks

“…The importance of objectives in view of the Pareto front has been analyzed in [19]. Part of the aforementioned study are the effects of system parameters such as the computational domain, and efficiency of adaptive loss weighting schemes [20,21]. In general, adaptive approaches have become popular in the optimization of PINNs [22,23], including adaptive activation functions [24,25] and soft attention mechanisms [26,27].…”