Arbitrary-Depth Universal Approximation Theorems for Operator Neural Networks

Annan, Yu,; Becquey, Chloé; Halikias, Diana; Mallory, Matthew Esmaili; Townsend, Alex

doi:10.48550/arxiv.2109.11354

Cited by 3 publications

(3 citation statements)

References 13 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…We also introduce various enhancements to DeepONet to accelerate its training and increase its accuracy, introducing for example the POD modes in the trunk net, obtained readily from the available training datasets. In addition to computational tests, we also perform a theoretical comparison of DeepONet versus FNO, following the published work on the theory of DeepONet in [2,24,25,26], and on the more recent theory of FNO in [4,27]. On this point, it is worth noted that DeepONet was based from the onset on the theorem of Chen & Chen [2], whereas the formulation of FNO was not theoretically justified originally, and the recent theoretical work covers only invariant kernels.…”

Section: Introductionmentioning

confidence: 99%

A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data

Lu,

Meng,

Cai

et al. 2021

Preprint

View full text Add to dashboard Cite

Neural operators can learn nonlinear mappings between function spaces and offer a new simulation paradigm for real-time prediction of complex dynamics for realistic diverse applications as well as for system identification in science and engineering. Herein, we investigate the performance of two neural operators, which have shown promising results so far, and we develop new practical extensions that will make them more accurate and robust and importantly more suitable for industrial-complexity applications. The first neural operator, DeepONet, was published in 2019 [1], and its original architecture was based on the universal approximation theorem of Chen & Chen [2]. The second one, named Fourier Neural Operator or FNO, was published in 2020 [3], and it is based on parameterizing the integral kernel in the Fourier space. DeepONet is represented by a summation of products of neural networks (NNs), corresponding to the branch NN for the input function and the trunk NN for the output function; both NNs are general architectures, e.g., the branch NN can be replaced with a CNN or a ResNet. According to [4], FNO in its continuous form can be viewed as a DeepONet with a specific architecture of the branch NN and a trunk NN represented by a trigonometric basis. In order to compare FNO with DeepONet for realistic setups, we develop several extensions of FNO that can deal with complex geometric domains as well as mappings where the input and output function spaces are of different dimensions. We also endow DeepONet with special features that provide inductive bias and accelerate training, and we present a faster implementation of DeepONet with cost comparable to the computational cost of FNO, which is based on the Fast Fourier Transform.We consider 16 different benchmarks to demonstrate the relative performance of the two neural operators, including instability wave analysis in hypersonic boundary layers, prediction of the vorticity field of a flapping airfoil, porous media simulations in complex-geometry domains, etc. The performance of DeepONet and FNO is comparable for relatively simple settings, but for complex geometries and especially noisy data, the performance of FNO deteriorates greatly. For example, for the instability wave analysis with only 0.1% noise added to the input data, the error of FNO increases 10,000 times making it inappropriate for such important applications, while there is hardly any effect of such noise on the DeepONet. This behavior of FNO may be related to an inherent mapping instability between function spaces, and hence future work should address this important issue. We also compare theoretically the two neural operators and obtain similar error estimates

show abstract

Section: Introductionmentioning

confidence: 99%

A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data

Lu,

Meng,

Cai

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…They have also proven that the DeepONet architecture has size O(|log( )| κ ) for any κ > 0 depending on the physical space dimension. In paper [46], the authors have shown that for non-polynomial activation functions, an operator with neural network of width five is arbitrarily close to any given continuous nonlinear operator. They have also shown the theoretical advantages of depth by constructing operator ReLU neural networks of depth 2k 3 + 8 with constant width, which they compare with other operator ReLU neural network of depth k.…”

Section: Neural Operator Theorymentioning

confidence: 99%

Physics-Informed Deep Neural Operator Networks

Goswami¹,

Bora²,

Yu³

et al. 2022

Preprint

View full text Add to dashboard Cite

Standard neural networks can approximate general nonlinear operators, represented either explicitly by a combination of mathematical operators, e.g., in an advection-diffusion-reaction partial differential equation, or simply as a black box, e.g., a system-of-systems. The first neural operator was the Deep Operator Network (DeepONet), proposed in 2019 based on rigorous approximation theory. Since then, a few other less general operators have been published, e.g., based on graph neural networks or Fourier transforms. For black box systems, training of neural operators is data-driven only but if the governing equations are known they can be incorporated into the loss function during training to develop physics-informed neural operators. Neural operators can be used as surrogates in design problems, uncertainty quantification, autonomous systems, and almost in any application requiring real-time inference. Moreover, independently pre-trained DeepONets can be used as components of a complex multi-physics system by coupling them together with relatively light training. Here, we present a review of DeepONet, the Fourier neural operator, and the graph neural operator, as well as appropriate extensions with feature expansions, and highlight their usefulness in diverse applications in computational mechanics, including porous media, fluid mechanics, and solid mechanics.

show abstract

“…Motivation: Operator learning techniques have recently attracted significant attention thanks to their effectiveness and favorable complexity in approximating maps between infinite-dimensional Banach spaces [1,2,3]. Techniques such as deep operator networks (DeepONets) [4], the family of neural operator methods [5], and operator-valued kernel methods [6,7] have demonstrated promise in building fast surrogate models for emulating complex physical processes, opening new avenues for sampling, inference and uncertainty quantification in very high-dimensional parameter spaces.…”

Section: Introductionmentioning

confidence: 99%

Improved architectures and training algorithms for deep operator networks

Wang¹,

Wang²,

Perdikaris³

2021

Preprint

View full text Add to dashboard Cite

Operator learning techniques have recently emerged as a powerful tool for learning maps between infinite-dimensional Banach spaces. Trained under appropriate constraints, they can also be effective in learning the solution operator of partial differential equations (PDEs) in an entirely selfsupervised manner. In this work we analyze the training dynamics of deep operator networks (DeepONets) through the lens of Neural Tangent Kernel (NTK) theory, and reveal a bias that favors the approximation of functions with larger magnitudes. To correct this bias we propose to adaptively re-weight the importance of each training example, and demonstrate how this procedure can effectively balance the magnitude of back-propagated gradients during training via gradient descent. We also propose a novel network architecture that is more resilient to vanishing gradient pathologies. Taken together, our developments provide new insights into the training of Deep-ONets and consistently improve their predictive accuracy by a factor of 10-50x, demonstrated in the challenging setting of learning PDE solution operators in the absence of paired input-output observations. All code and data accompanying this manuscript will be made publicly available at https://github.com/PredictiveIntelligenceLab/ImprovedDeepONets.

show abstract

Arbitrary-Depth Universal Approximation Theorems for Operator Neural Networks

Cited by 3 publications

References 13 publications

A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data

A comprehensive and fair comparison of two neural operators (with practical extensions) based on FAIR data

Physics-Informed Deep Neural Operator Networks

Improved architectures and training algorithms for deep operator networks

Contact Info

Product

Resources

About