Learning and Retrieval Operational Modes for Three-Layer Restricted Boltzmann Machines

Abstract

We consider a three-layer restricted Boltzmann machine, where the two visible layers (encoding for input and output, respectively) are made of binary neurons while the hidden layer is made of Gaussian neurons, and we show a formal equivalence with a Hopfield model. The machine architecture allows for different learning and operational modes: when all neurons are free to evolve we recover a standard Hopfield model whose size corresponds to the overall size of visible neurons; when input neurons are clamped we recover a Hopfield model, whose size corresponds to the size of the output layer, endowed with an external field as well as additional slow noise. The former stems from the signal provided by the input layer and tends to favour retrieval, the latter can be related to the statistical properties of the training set and tends to impair the retrieval performance of the network. We address this model by rigorous techniques, finding an explicit expression for its free-energy, whence a phase-diagram showing the performance of the system as parameters are tuned.

Replica symmetry breaking in supervised and unsupervised Hebbian networks

“…For example, the Hopfield model has been shown to be equivalent to a restricted Boltzmann machine [21], an archetypical model for machine learning [22], and sparse restricted Boltzmann machines have been mapped to Hopfield models with diluted patterns [23,24]. Furthermore, restricted Boltzmann machines with generic priors have led to the definition of generalized Hopfield models [25][26][27] and neural networks with multi-node Hebbian interactions have recently been shown to be equivalent to higher-order Boltzmann machines [28,29] and deep Boltzmann machines [30,31]. As a result, multi-node Hebbian learning is receiving a second wave of interest since its foundation in the eighties [13,14] as a paradigm to understand deep learning [17,32].…”

A transport equation approach for deep neural networks with quenched random weights

“…Also motivated by the impressive successes obtained in artificial intelligence via deep learning methods (which, beyond DBMs, include a number of other different neural networks), this kind of structures have recently attracted a wide interest (see e.g. [12][13][14][15][16][17][18][19]). Conceptually, training algorithms for Boltzmann machines as well as their deep versions are based on the minimization of a fixed objective function (e.g.…”