Optimal storage of a neural network model: a replica symmetry-breaking solution

Untitled

“…It has been shown that beyond capacity the replica symmetric solution is thermodynamically unstable [28]. One [22,23] and two [19] step replica symmetry breaking solutions were presented, while Ref. [19] proved that no finite step symmetry breaking ansatz can possibly be thermodynamically stable.…”

“…The inspection of the first few R = 0, 1, 2 cases [4,22,23,19] allows, in the spirit of Parisi's [29], the generalization of the energy term (2.13d) in the replica free energy to arbitrary R as…”

“…Thus the description of a single neuron beyond its storage capacity is of importance also from the viewpoint of networked neurons. Furthermore, a close analogy exists between the behavior of model neurons beyond capacity and the glassy, frustrated, phase of disordered spin systems [6,7,22,23,19,24]. Therefore, the understanding of the way a single neuron works may have ramifications beyond the field of artificial neural networks.…”

Threshold-induced phase transitions in perceptrons

“…The calculation is similar to 4,5] for real valued weights and a spherical constraint and to 6] for binary weights, i.e., an Ising constraint; however, we allow for a threshold and biased output and input distributions. In the following the real valued weight boolean perceptron will be referred to as the spherical (boolean) perceptron, whereas the binary valued weight boolean perceptron will be referred to as the Ising (boolean) perceptron.…”

“…While most of the research concentrated on exploring the learning ability and network capacity below saturation (for a review see 2,3] and references therein), we will concentrate in this paper on the errors of a boolean perceptron above its saturation limit, or capacity limit c , working within a replica framework. Earlier studies 4,5,6] have examined particularly the cases of zero stability of the stored patterns, the e ect of di erent error functions on the error rates, and the distribution of pattern stabilities. Here, we will extend this work by allowing for a threshold and biased input and output distributions and investigate both real valued (spherical constraint) and binary weights (Ising constraint).…”