A transport equation approach for deep neural networks with quenched random weights

Agliari, Elena; Albanese, Linda; Alemanno, Francesco; Fachechi, Alberto

doi:10.1088/1751-8121/ac38ec

Cited by 9 publications

(3 citation statements)

References 40 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…The latter, accounting for RSB phenomena, can be obtained by iteratively perturbing the RS interpolation scheme (e.g. see [47,37,48]), thus, our results find direct application on the practical side and provide the starting point for further refinements on the theoretical side.…”

Section: Generalities and Notationmentioning

confidence: 79%

“…In this case, we can take advantage of rigorous mathematical methods by applying sum rules [26] or by mapping the relevant quantities (the free energy or the model order parameters) of the statistical setting to the solutions of PDE systems. Indeed, differential equations involving the partition functions (or related quantities) of thermodynamic models have been extensively investigated in the literature, see for example [27][28][29][30][31][32][33][34][35][36][37]. In particular, they allow us to express the equation of state (or the self-consistency equations) governing the equilibrium dynamics of the system in terms of solutions of non-linear differential equations, and to describe phase transition phenomena as the development of shock waves, thus linking critical behaviours to gradient catastrophe theory [38][39][40][41].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Non-linear PDEs approach to statistical mechanics of Dense Associative Memories

Agliari¹,

Fachechi²,

Marullo³

2022

Preprint

View full text Add to dashboard Cite

Dense associative memories (DAM), are widespread models in artificial intelligence used for pattern recognition tasks; computationally, they have been proven to be robust against adversarial input and theoretically, leveraging their analogy with spin-glass systems, they are usually treated by means of statistical-mechanics tools. Here we develop analytical methods, based on nonlinear PDEs, to investigate their functioning. In particular, we prove differential identities involving DAM's partition function and macroscopic observables useful for a qualitative and quantitative analysis of the system. These results allow for a deeper comprehension of the mechanisms underlying DAMs and provide interdisciplinary tools for their study.

show abstract

Section: Generalities and Notationmentioning

confidence: 79%

Section: Introductionmentioning

confidence: 99%

Non-linear PDEs approach to statistical mechanics of Dense Associative Memories

Agliari¹,

Fachechi²,

Marullo³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For example, the Hopfield model has been shown to be equivalent to a restricted Boltzmann machine [21], an archetypical model for machine learning [22], and sparse restricted Boltzmann machines have been mapped to Hopfield models with diluted patterns [23,24]. Furthermore, restricted Boltzmann machines with generic priors have led to the definition of generalized Hopfield models [25][26][27] and neural networks with multi-node Hebbian interactions have recently been shown to be equivalent to higher-order Boltzmann machines [28,29] and deep Boltzmann machines [30,31]. As a result, multi-node Hebbian learning is receiving a second wave of interest since its foundation in the eighties [13,14] as a paradigm to understand deep learning [17,32].…”

Section: Introductionmentioning

confidence: 99%

Replica symmetry breaking in supervised and unsupervised Hebbian networks

Albanese,

Alessandrelli,

Annibale

et al. 2024

J. Phys. A: Math. Theor.

View full text Add to dashboard Cite

Hebbian neural networks with multi-node interactions, often called Dense Associative Memories, have recently attracted considerable interest in the statistical mechanics community, as they have been shown to outperform their pairwise counterparts in a number of features, including resilience against adversarial attacks, pattern retrieval with extremely weak signals and supra-linear storage capacities. However, their analysis has so far been carried out within a replica-symmetric theory. In this manuscript, we relax the assumption of replica symmetry and analyse these systems at one step of replica-symmetry breaking, focusing on two different prescriptions for the interactions that we will refer to as supervised and unsupervised learning. We derive the phase diagram of the model using two different approaches, namely Parisi's hierarchical ansatz for the relationship between different replicas within the replica approach, and the so-called telescope ansatz within Guerra's interpolation method: our results show that replica-symmetry breaking does not alter the threshold for learning and slightly increases the maximal storage capacity. Further, we also derive analytically the instability line of the replica-symmetric theory, using a generalization of the De Almeida and Thouless approach.

show abstract

Learning and Retrieval Operational Modes for Three-Layer Restricted Boltzmann Machines

Agliari

Sebastiani

2021

J Stat Phys

View full text Add to dashboard Cite

We consider a three-layer restricted Boltzmann machine, where the two visible layers (encoding for input and output, respectively) are made of binary neurons while the hidden layer is made of Gaussian neurons, and we show a formal equivalence with a Hopfield model. The machine architecture allows for different learning and operational modes: when all neurons are free to evolve we recover a standard Hopfield model whose size corresponds to the overall size of visible neurons; when input neurons are clamped we recover a Hopfield model, whose size corresponds to the size of the output layer, endowed with an external field as well as additional slow noise. The former stems from the signal provided by the input layer and tends to favour retrieval, the latter can be related to the statistical properties of the training set and tends to impair the retrieval performance of the network. We address this model by rigorous techniques, finding an explicit expression for its free-energy, whence a phase-diagram showing the performance of the system as parameters are tuned.

show abstract

A transport equation approach for deep neural networks with quenched random weights

Cited by 9 publications

References 40 publications

Non-linear PDEs approach to statistical mechanics of Dense Associative Memories

Non-linear PDEs approach to statistical mechanics of Dense Associative Memories

Replica symmetry breaking in supervised and unsupervised Hebbian networks

Learning and Retrieval Operational Modes for Three-Layer Restricted Boltzmann Machines

Contact Info

Product

Resources

About