Learning Physics-Informed Neural Networks without Stacked Back-propagation

Abstract

Physics-Informed Neural Network (PINN) has become a commonly used machine learning approach to solve partial differential equations (PDE). But, facing highdimensional second-order PDE problems, PINN will suffer from severe scalability issues since its loss includes second-order derivatives, the computational cost of which will grow along with the dimension during stacked back-propagation. In this paper, we develop a novel approach that can significantly accelerate the training of Physics-Informed Neural Networks. In particular, we parameterize the PDE solution by the Gaussian smoothed model and show that, derived from Stein's Identity, the second-order derivatives can be efficiently calculated without backpropagation. We further discuss the model capacity and provide variance reduction methods to address key limitations in the derivative estimation. Experimental results show that our proposed method can achieve competitive error compared to standard PINN training but is two orders of magnitude faster.

Does Federated Learning Really Need Backpropagation?

“…Gradient-based optimization techniques (either first-order or higher-order) are the most frequently used tools to train deep networks (Goodfellow et al, 2016). Nevertheless, recent progress demonstrates promising applications of zero-order optimization methods for training, particularly when exact derivatives cannot be obtained (Flaxman et al, 2004;Nesterov & Spokoiny, 2017;Liu et al, 2020a) or backward processes are computationally pro-hibitive (Pang et al, 2020;He et al, 2022). Zero-order approaches require only multiple forward processes that may be executed in parallel.…”

“…Efficiency in FL. It is widely understood that the communication and computational efficiency is a primary bottleneck for deploying FL in practice (Wang et al, 2019b;Rothchild et al, 2020;Chen et al, 2021;Balakrishnan et al, 2022;Wang et al, 2022). Specifically, communicating between the server and clients could be potentially expensive and unreliable.…”