Understanding neural networks with reproducing kernel Banach spaces

Bartolucci, Francesca; Vito, Ernesto De; Rosasco, Lorenzo; Vigogna, Stefano

doi:10.1016/j.acha.2022.08.006

Cited by 14 publications

(16 citation statements)

References 44 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…It would then be of practical interest to study the fluctuation and concentration properties of such zero-mean noise term with respect to the size of the neural network. Other approximation methods for the MZ memory and noise terms can be built on the backbone of continuous-time approximation methods, e.g., based on functional analysis [53,54,29], combinatorics [55], data-driven methods [3,40,30,33], Markovian embedding techniques [21,18,32,25,7], or projections based on reproducing kernel Hilbert or Banach spaces [1,39,52].…”

Section: Discussionmentioning

confidence: 99%

“…Other cases where M is not square can be handled similarly [38,22]. By following the same procedure, it is possible also to derive the transition density for the general neural network model (1).…”

Section: Multi-layer Conditional Transition Densitymentioning

confidence: 99%

“…In the context of deep learning, the MZ formalism can be used to reduce the total number of degrees of freedom of the neural network, e.g., by reducing the number of neurons per layer (using projection operators), or by transforming deep networks into shallows networks, e.g., using some approximation of the MZ memory operator 1 . Computing the solution of the MZ equation for deep learning is not an easy task.…”

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

The Mori-Zwanzig formulation of deep learning

Venturi¹,

Li²

2022

Preprint

View full text Add to dashboard Cite

We develop a new formulation of deep learning based on the Mori-Zwanzig (MZ) formalism of irreversible statistical mechanics. The new formulation is built upon the well-known duality between deep neural networks and discrete stochastic dynamical systems, and it allows us to directly propagate quantities of interest (conditional expectations and probability density functions) forward and backward through the network by means of exact linear operator equations. Such new equations can be used as a starting point to develop new effective parameterizations of deep neural networks, and provide a new framework to study deep-learning via operator theoretic methods. The proposed MZ formulation of deep learning naturally introduces a new concept, i.e., the memory of the neural network, which plays a fundamental role in low-dimensional modeling and parameterization. By using the theory of contraction mappings, we develop sufficient conditions for the memory of the neural network to decay with the number of layers. This allows us to rigorously transform deep networks into shallow ones, e.g., by reducing the number of neurons per layer (using projection operators), or by reducing the total number of layers (using the decay property of the memory operator).

show abstract

Section: Discussionmentioning

confidence: 99%

Section: Multi-layer Conditional Transition Densitymentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

The Mori-Zwanzig formulation of deep learning

Venturi¹,

Li²

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…With the corresponding norms, they are able to prove bounds on the generalisation error. On the other hand Bartolucci et al [2023] used reproducing kernel Banach spaces to understand neural networks and prove a general reproducing theorem for these spaces. Nowak [2020, 2021a,b] analysed the neural network using ridge splines, where the norm can be represented in terms of an inverse Radon transform of a bounded real measure.…”

Section: Neural Networkmentioning

confidence: 99%

“…Nowak [2020, 2021a,b] analysed the neural network using ridge splines, where the norm can be represented in terms of an inverse Radon transform of a bounded real measure. Bartolucci et al [2023] showed that this method is a constructive realisation of a reproducing kernel Banach space for certain activation functions like the ReLu.…”

Section: Neural Networkmentioning

confidence: 99%

Analysis of dynamics of neural fields and neural networks

Spek¹

View full text Add to dashboard Cite

We consider abstract delay differential equations which take values in arbitrary Banach spaces. Here the right-hand side can be broken down into a bounded, non-linear, compact operator which depends on the delay and a linear differential operator which does not need to be bounded. We extend the results of Janssens [2019] on the sun-star calculus for these equations, specifically we describe the generator of the semigroup corresponding to the linearisation of the delay equation. One key result is that the space of continuous functions on a bounded interval taking values in some arbitrary Banach space Y is sun-reflexive with respect to the shift semigroup if and only if Y itself is reflexive. Finally, we develop a method to analytically compute the normal form coefficients of certain bifurcations for these abstract delay differential equations.

show abstract

Reproducing kernel Hilbert spaces cannot contain all continuous functions on a compact metric space

Steinwart

2024

Arch. Math.

View full text Add to dashboard Cite

show abstract

Understanding neural networks with reproducing kernel Banach spaces

Cited by 14 publications

References 44 publications

The Mori-Zwanzig formulation of deep learning

The Mori-Zwanzig formulation of deep learning

Analysis of dynamics of neural fields and neural networks

Reproducing kernel Hilbert spaces cannot contain all continuous functions on a compact metric space

Contact Info

Product

Resources

About