Error estimates for DeepONets: a deep learning framework in infinite dimensions

Abstract

DeepONets have recently been proposed as a framework for learning nonlinear operators mapping between infinite-dimensional Banach spaces. We analyze DeepONets and prove estimates on the resulting approximation and generalization errors. In particular, we extend the universal approximation property of DeepONets to include measurable mappings in non-compact spaces. By a decomposition of the error into encoding, approximation and reconstruction errors, we prove both lower and upper bounds on the total error, relating it to the spectral decay properties of the covariance operators, associated with the underlying measures. We derive almost optimal error bounds with very general affine reconstructors and with random sensor locations as well as bounds on the generalization error, using covering number arguments. We illustrate our general framework with four prototypical examples of nonlinear operators, namely those arising in a nonlinear forced ordinary differential equation, an elliptic partial differential equation (PDE) with variable coefficients and nonlinear parabolic and hyperbolic PDEs. While the approximation of arbitrary Lipschitz operators by DeepONets to accuracy $\epsilon $ is argued to suffer from a ‘curse of dimensionality’ (requiring a neural networks of exponential size in $1/\epsilon $), in contrast, for all the above concrete examples of interest, we rigorously prove that DeepONets can break this curse of dimensionality (achieving accuracy $\epsilon $ with neural networks of size that can grow algebraically in $1/\epsilon $).Thus, we demonstrate the efficient approximation of a potentially large class of operators with this machine learning framework.

On universal approximation and error bounds for Fourier Neural Operators

“…Furthermore, the operator G may not always be continuous. To address these issues, one can follow the recent paper [15], where the authors prove a more general version for the universal approximation of operators for DeepOnets; for any input measure µ, and a Borel measurable operator G, such that…”

“…More recently, the authors of [21] have proposed using deep, instead of shallow, neural networks in both the trunk and branch net and have christened the resulting architecture as a DeepOnet. In a recent article [15], the universal approximation property of DeepOnets was extended, making it completely analogous to universal approximation results for finite-dimensional functions by neural networks. The authors of [15] were also able to show that DeepOnets can break the curse of dimensionality for a large variety of PDE learning tasks.…”

“…In a recent article [15], the universal approximation property of DeepOnets was extended, making it completely analogous to universal approximation results for finite-dimensional functions by neural networks. The authors of [15] were also able to show that DeepOnets can break the curse of dimensionality for a large variety of PDE learning tasks. Hence, in spite of the underlying infinite-dimensional setting, DeepOnets are capable of approximating a large variety of nonlinear operators efficiently.…”

“…In particular, it is unclear if neural operators such as FNOs are universal i.e., if they can approximate a large class of nonlinear infinite-dimensional operators. Moreover in this infinite-dimensional setting, universality does not suffice to indicate computational viability or efficiency as the size of the underlying neural networks might grow exponentially with respect to increasing accuracy, see discussion in [15] on this issue. Hence in addition to universality, it is natural to ask if neural operators can efficiently approximate a large class of operators, such as those arising in the simulation of parametric PDEs.…”

Physics Informed Neural Networks (PINNs)for approximating nonlinear dispersive PDEs

“…A very incomplete list of examples where deep learning is used for the numerical solutions of differential equations includes the solution of high-dimensional linear and semi-linear parabolic partial differential equations [9,13] and references therein, and for many-query problems such as those arising in uncertainty quantification (UQ), PDE constrained optimization and (Bayesian) inverse problems. Such problems can be recast as parametric partial differential equations and the use of deep neural networks in their solution is explored for elliptic and parabolic PDEs in [22,43], for transport PDEs [24] and for hyperbolic and related PDEs [6,[34][35][36], and as operator learning frameworks in [2,28,30,32] and references therein. All the afore-mentioned methods are of the supervised learning type [12] i.e., the underlying deep neural networks have to be trained on data, either available from measurements or generated by numerical simulations.…”