A multiscale neural network based on hierarchical matrices

Fan, Yuwei; Lin, Lin; Ying, Lexing; Zepeda-Núñez, Leonardo

doi:10.48550/arxiv.1807.01883

Cited by 26 publications

(42 citation statements)

References 49 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, DNN-based approaches have been actively explored for a variety of scientific computing problems, e.g., solving high-dimensional partial differential equations (E et al, 2017;Khoo et al, 2017;He et al, 2018;Fan et al, 2018) and molecular dynamics (MD) simulations . However, the behaviors of DNNs applied to these problems are not well-understood.…”

Section: F-principle In Solving Differential Equationmentioning

confidence: 99%

Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

Xu,

Zhang,

Luo

et al. 2019

Preprint

114

View full text Add to dashboard Cite

We study the training process of Deep Neural Networks (DNNs) from the Fourier analysis perspective. We demonstrate a very universal Frequency Principle (F-Principle) -DNNs often fit target functions from low to high frequencies -on high-dimensional benchmark datasets such as MNIST/CIFAR10 and deep neural networks such as VGG16. This F-Principle of DNNs is opposite to the behavior of most conventional iterative numerical schemes (e.g., Jacobi method), which exhibit faster convergence for higher frequencies for various scientific computing problems. With a simple theory, we illustrate that this F-Principle results from the regularity of the commonly used activation functions. The F-Principle implies an implicit bias that DNNs tend to fit training data by a low-frequency function. This understanding provides an explanation of good generalization of DNNs on most real datasets and bad generalization of DNNs on parity function or randomized dataset.

show abstract

Section: F-principle In Solving Differential Equationmentioning

confidence: 99%

Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

Xu,

Zhang,

Luo

et al. 2019

Preprint

114

View full text Add to dashboard Cite

show abstract

“…They have demonstrated advantages in both efficiency and accuracy compared to conventional solvers. They are even able to tackle previously intractable problems such as higher-dimensional, multiscale, high-contrast, and chaotic PDE systems [33,3,8,22,12,2]. Broadly, ML-based approaches for PDEs can be divided into two categories: optimizing to solve for a specific solution function of PDE vs. learning the solution operator over a family of PDEs.…”

Section: Introductionmentioning

confidence: 99%

Physics-Informed Neural Operator for Learning Partial Differential Equations

Li¹,

Zheng²,

Kovachki³

et al. 2021

Preprint

View full text Add to dashboard Cite

Machine learning methods have recently shown promise in solving partial differential equations (PDEs). They can be classified into two broad categories: approximating the solution function and learning the solution operator. The Physics-Informed Neural Network (PINN) is an example of the former while the Fourier neural operator (FNO) is an example of the latter. Both these approaches have shortcomings. The optimization in PINN is challenging and prone to failure, especially on multi-scale dynamic systems. FNO does not suffer from this optimization issue since it carries out supervised learning on a given dataset, but obtaining such data may be too expensive or infeasible. In this work, we propose the physics-informed neural operator (PINO), where we combine the operating-learning and function-optimization frameworks. This integrated approach improves convergence rates and accuracy over both PINN and FNO models. In the operator-learning phase, PINO learns the solution operator over multiple instances of the parametric PDE family. In the test-time optimization phase, PINO optimizes the pre-trained operator ansatz for the querying instance of the PDE. Experiments show PINO outperforms previous ML methods on many popular PDE families while retaining the extraordinary speed-up of FNO compared to solvers. In particular, PINO accurately solves long temporal transient flows and Kolmogorov flows.

show abstract

“…Deep learning has achieved great success as in many fields (LeCun et al, 2015), e.g., speech recognition (Amodei et al, 2016), object recognition (Eitel et al, 2015), natural language processing (Young et al, 2018) and computer game control (Mnih et al, 2015). It has also been adopted into algorithms to solve scientific computing problems (E et al, 2017;Khoo et al, 2017;He et al, 2018;Fan et al, 2018). In principle, the universal approximation theorem states that a commonly-used Deep Neural Network (DNN) of sufficiently large width can approximate any function to a desired precision (Cybenko, 1989).…”

Section: Introductionmentioning

confidence: 99%

Theory of the Frequency Principle for General Deep Neural Networks

Luo¹,

Ma²,

Xu³

et al. 2019

Preprint

View full text Add to dashboard Cite

Along with fruitful applications of Deep Neural Networks (DNNs) to realistic problems, recently, some empirical studies of DNNs reported a universal phenomenon of Frequency Principle (F-Principle): a DNN tends to learn a target function from low to high frequencies during the training. The F-Principle has been very useful in providing both qualitative and quantitative understandings of DNNs. In this paper, we rigorously investigate the F-Principle for the training dynamics of a general DNN at three stages: initial stage, intermediate stage, and final stage. For each stage, a theorem is provided in terms of proper quantities characterizing the F-Principle. Our results are general in the sense that they work for multilayer networks with general activation functions, population densities of data, and a large class of loss functions. Our work lays a theoretical foundation of the F-Principle for a better understanding of the training process of DNNs.

show abstract

A multiscale neural network based on hierarchical matrices

Cited by 26 publications

References 49 publications

Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networks

Physics-Informed Neural Operator for Learning Partial Differential Equations

Theory of the Frequency Principle for General Deep Neural Networks

Contact Info

Product

Resources

About