On Universal Approximation by Neural Networks with Uniform Guarantees on Approximation of Infinite Dimensional Maps

Abstract

The study of universal approximation of arbitrary functions f : X → Y by neural networks has a rich and thorough history dating back to Kolmogorov (1957). In the case of learning finite dimensional maps, many authors have shown various forms of the universality of both fixed depth and fixed width neural networks. However, in many cases, these classical results fail to extend to the recent use of approximations of neural networks with infinitely many units for functional data analysis, dynamical systems identification, and other applications where either X or Y become infinite dimensional. Two questions naturally arise: which infinite dimensional analogues of neural networks are sufficient to approximate any map f : X → Y, and when do the finite approximations to these analogues used in practice approximate f uniformly over its infinite dimensional domain X ?In this paper, we answer the open question of universal approximation of nonlinear operators when X and Y are both infinite dimensional. We show that for a large class of different infinite analogues of neural networks, any continuous map can be approximated arbitrarily closely with some mild topological conditions on X . Additionally, we provide the first lower-bound on the minimal number of input and output units required by a finite approximation to an infinite neural network to guarantee that it can uniformly approximate any nonlinear operator using samples from its inputs and outputs.

Universal Approximation Theorem for Equivariant Maps by Group CNNs

“…We provide proof of Theorem 13 in the appendix. We note that S ⊂ R d and T ⊂ R d ′ in Theorem 13 are allowed to be non-compact unlike the result in Guss and Salakhutdinov (2019). Combining Theorem 9 with Theorem 13, we obtain the following theorem.…”

“…After his work, some researchers showed similar results to generalize the sigmoidal function to a larger class of activation functions as Barron (1994), Hornik et al (1989), Funahashi (1989), Kůrková (1992) and Sonoda and Murata (2017). These results were approximations to functional representations between finite-dimensional vector spaces, but recently Guss and Salakhutdinov (2019) generalized them to continuous maps between infinite-dimensional function spaces in Guss and Salakhutdinov (2019).…”

“…µ φ ≪ ν means that µ φ is absolutely continuous with respect to ν 8. A singleton is a set with exactly one element 9. The upper half plane is defined by H d+1 := {(x1, .…”

Fast Resolution Agnostic Neural Techniques to Solve Partial Differential Equations

“…Neural operator is an emerging deep learning technique that is different from a typical neural network in several ways. While neural networks are able to approximate any function, which is a map between finite-dimensional spaces Cybenko (1989), to approximate an operator, which is a map between infinite dimensional spaces, a network of infinite length is required Guss & Salakhutdinov (2019). Thus, neural networks fail to accurately approximate solution operators of PDEs, which are mappings between infinite dimensional spaces.…”