Derivative-Informed Neural Operator: An Efficient Framework for High-Dimensional Parametric Derivative Learning

Abstract

Neural operators have gained significant attention recently due to their ability to approximate high-dimensional parametric maps between function spaces. At present, only parametric function approximation has been addressed in the neural operator literature. In this work we investigate incorporating parametric derivative information in neural operator training; this information can improve function approximations, additionally it can be used to improve the approximation of the derivative with respect to the parameter, which is often the key to scalable solution of high-dimensional outer-loop problems (e.g. Bayesian inverse problems). Parametric Jacobian information is formally intractable to incorporate due to its high-dimensionality, to address this concern we propose strategies based on reduced SVD, randomized sketching and the use of reduced basis surrogates. All of these strategies only require only O(r) Jacobian actions to construct sample Jacobian data, and allow us to reduce the linear algebra and memory costs associated with the Jacobian training from the product of the input and output dimensions down to O(r 2 ), where r is the dimensionality associated with the dimension reduction technique.Numerical results for parametric PDE problems demonstrate that the addition of derivative information to the training problem can significantly improve the parametric map approximation, particularly given few data. When Jacobian actions are inexpensive compared to the parametric map, this information can be economically substituted for parametric map data. Additionally we show that Jacobian error approximations improve significantly with the introduction of Jacobian training data. This result opens the door to the use of derivative-informed neural operators (DINOs) in outer-loop algorithms where they can amortize the additional training data cost via repeated evaluations.

Residual-based error correction for neural operator accelerated infinite-dimensional Bayesian inverse problems

“…where the choice of p depends on the regularity of the forward operator F. The choice p = 2 is often taken in practice, and additional learning of derivatives may be included via generalizing to T = W 1,p (M, ν M ; U), as in [71].…”

“…In order to address this issue, the input and output spaces of neural operators are often restricted to some finite-dimensional reduced bases of M and U. Different architectures of neural operator incorporate different classical reduced basis representations such as proper orthogonal decomposition (POD) [33,34,39,40], Fourier representation [36], multipole graph representations [37], and derivative sensitivity bases [39,40,71] among others. These reduced basis representations offer a scalable means of learning structured maps between infinite-dimensional spaces, by taking advantage of their compact representations in specific bases, if such representations exist.…”

“…However, we observe substantially the performance improvement of neural operators trained with this optimizer. Moreover, unlike [71] this network does not include derivative approximations during the formulation of the operator learning problem and training, and the derivative information is only used in the architecture. For more information on the neural network architectures employed in the two numerical examples, see [39,40].…”

“…In most cases, the loss function of neural operator training is uninformed of the gradient of forward operator, leading to poor performances in gradient estimations. See[71] for how to incorporate high-dimensional derivative information in neural operator training.…”

A Greedy Sensor Selection Algorithm for Hyperparameterized Linear Bayesian Inverse Problems

“…In particular, [21,22] use similar algorithmic approaches to this work by applying a greedy algorithm to maximize the expected information gain. Common strategies for dealing with the high dimensions imposed by the PDE model use the framework in [24] for discretization, combined with parameter reduction methods (e.g., [25,26,27,28,29,30,31]) and model order reduction (MOR) methods for uncertainty quantification (UQ) problems (e.g., [32,33,34,35,36]).…”