The edge of chaos: quantum field theory and deep neural networks

Jefferson, Ro

doi:10.21468/scipostphys.12.3.081

Cited by 14 publications

(13 citation statements)

References 45 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…In [76], [36] a completely different approach for the correspondence between Euclidean QFTs and NNs is presented. Starting with a stochastic differential equation, which plays the role of a master equation for the neural network, the authors construct an action and a path integral, which provides the QFT attached to the network.…”

Section: Discussionmentioning

confidence: 99%

See 1 more Smart Citation

p-Adic Statistical Field Theory and Deep Belief Networks

Zúñiga–Galindo¹

2022

Preprint

View full text Add to dashboard Cite

In this work we initiate the study of the correspondence between p-adic statistical field theories (SFTs) and neural networks (NNs). In general quantum field theories over a p-adic spacetime can be formulated in a rigorous way. Nowadays these theories are considered just mathematical toy models for understanding the problems of the true theories. In this work we show these theories are deeply connected with the deep belief networks (DBNs). Hinton et al. constructed DBNs by stacking several restricted Boltzmann machines (RBMs). The purpose of this construction is to obtain a network with a hierarchical structure (a deep learning architecture). An RBM corresponds a certain spin glass, thus a DBN should correspond to an ultrametric (hierarchical) spin glass. A model of such system can be easily constructed by using p-adic numbers. In our approach, a p-adic SFT corresponds to a p-adic continuous DBN, and a discretization of this theory corresponds to a p-adic discrete DBN. We show that these last machines are universal approximators. In the p-adic framework, the correspondence between SFTs and NNs is not fully developed. We point out several open problems..

show abstract

Section: Discussionmentioning

confidence: 99%

“…The case in which the architecture is embedded in a stochastic differential equation or in a discrete mater equation has been studied intensively lately, see e.g. [9], [36], [40], see also [17], [21]. The p-adic counterpart of this correspondence is an open problem.…”

Section: Discussionmentioning

confidence: 99%

p-Adic Statistical Field Theory and Deep Belief Networks

Zúñiga–Galindo¹

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…which is meant as a 2 × 2 tensor equation consisting of tensors with four time indices (see appendix A.5 for details). Due to the normalization W aux (0|C) = 0, given by ( 30) or (29), respectively, the Hessian has a zero in the upper left corner S…”

Section: Next-to-leading Order Correctionsmentioning

confidence: 99%

“…The Gaussian process thus has limited capability of quantifying the expressivity of neural networks in relation to the required resources, such as the number of trained weights. Studies on finite-size corrections beyond the n → ∞ limit are so far restricted to DNNs [17][18][19][20][21][22][23][24][25][26][27][28] (but see [29] for stationary continuous-time recurrent networks). Understanding the limits of the putative equivalence of DNNs and RNNs on the mean-field level requires a common theoretical basis for the two architectures that would extend to finite n and finite n .…”

Section: Introductionmentioning

confidence: 99%

Unified field theoretical approach to deep and recurrent neuronal networks

et al. 2022

View full text Add to dashboard Cite

Understanding capabilities and limitations of different network architectures is of fundamental importance to machine learning. Bayesian inference on Gaussian processes has proven to be a viable approach for studying recurrent and deep networks in the limit of infinite layer width, n → ∞. Here we present a unified and systematic derivation of the mean-field theory for both architectures that starts from first principles by employing established methods from statistical physics of disordered systems. The theory elucidates that while the mean-field equations are different with regard to their temporal structure, they yet yield identical Gaussian kernels when readouts are taken at a single time point or layer, respectively. Bayesian inference applied to classification then predicts identical performance and capabilities for the two architectures. Numerically, we find that convergence towards the mean-field theory is typically slower for recurrent networks than for deep networks and the convergence speed depends non-trivially on the parameters of the weight prior as well as the depth or number of time steps, respectively. Our method exposes that Gaussian processes are but the lowest order of a systematic expansion in 1/n and we compute next-to-leading-order corrections which turn out to be architecture-specific. The formalism thus paves the way to investigate the fundamental differences between recurrent and deep architectures at finite widths n.

show abstract

“…In appropriate limits, these networks become Gaussian processes [13][14][15][16][17]. Such Gaussian processes are closely related to the situation when one wants to understand the ensemble of quantum fluctuations of scalar field theories as discussed in [18][19][20][21][22]. Related to our work, we utilise the same interpretation of a NN as a scalar field.…”

Section: Related Workmentioning

confidence: 99%

A duality connecting neural network and cosmological dynamics

Krippendorf¹,

Spannowsky

2022

Mach. Learn.: Sci. Technol.

View full text Add to dashboard Cite

We demonstrate that the dynamics of neural networks trained with gradient descent and the dynamics of scalar fields in a flat, vacuum energy dominated Universe are structurally profoundly related. This duality provides the framework for synergies between these systems, to understand and explain neural network dynamics and new ways of simulating and describing early Universe models. Working in the continuous-time limit of neural networks, we analytically match the dynamics of the mean background and the dynamics of small perturbations around the mean field, highlighting potential differences in separate limits. We perform empirical tests of this analytic description and quantitatively show the dependence of the effective field theory parameters on hyperparameters of the neural network. As a result of this duality, the cosmological constant is matched inversely to the learning rate in the gradient descent update.

show abstract

The edge of chaos: quantum field theory and deep neural networks

Cited by 14 publications

References 45 publications

p-Adic Statistical Field Theory and Deep Belief Networks

p-Adic Statistical Field Theory and Deep Belief Networks

Unified field theoretical approach to deep and recurrent neuronal networks

A duality connecting neural network and cosmological dynamics

Contact Info

Product

Resources

About