Associative recall of memory without errors

Abstract

A neural network which is capable of recalling without errors any set of linearly independent patterns is studied. The network is based on a Hamiltonian version of the model of Personnaz et al.The energy of a state of N (+1}neurons is the square of the Euclidean distancein phase spacebetween the state and the linear subspace spanned by the patterns. This energy corresponds to nonlocal updatings of the synapses in the learning mode. Results of the mean-field theory (MFT) of the system as well as computer simulations are presented. The stable and metastable states of the network are studied as a function of "temperature" T and a=p/X, where p is the number of ernbedded patterns. The maximum capacity of the network is a=1. For all a (0&a&1}the embedded patterns are not only locally stable but are global minima of the energy. The patterns appear, as metastable states, below a temperature T =T~(a). The temperature T~(a) decreases to zero as a~1. The spurious states of the network are studied in detail in the case of random uncorrelated patterns. At finite p, they are identical to the mixture states of Hopfield s model. At finite a, a spin-glass phase exists as a metastable state. According to the replica symmetric MFT the spinglass state becomes degenerate with the patterns at a=ag:1 2/~and disappears above it. Possible interpretations of this unusual result are discussed. The average radius of attraction R of the patterns has been determined by computer simulations, for sizes up to N =400. The value of R for 0&a & 1 depends on the details of the dynamics. Results for both parallel and serial dynamics are presented. In both cases R is unity (the largest distance in phase space by. definition} at a~0 and decreases monotonically to zero as a~1. -Contrary to the MFT, simulations have not revealed, so far, any singularity in the properties of the spurious states at an intermediate value of a.

Optimal learning rules for familiarity detection

“…For instance, the pseudo-inverse rule is has a higher capacity (1 bit per synapse), but it is neither linear, nor local (Kanter and Sompolinsky (1987); Hertz et al (1991)). …”

The roles of stability and symmetry in the dynamics of neural networks

“…Outstanding popularity was gained by the Hopfield model (Hopfield 1982, Amit 1987 which, by the symmetry of its interactions and by its Monte Carlo dynamics, has had great appeal for physicists trained in statistical mechanics. Examples have shown, however, that similar behaviour may be reached with quite different architectures and philosophies, such as the asymmetric strict-stability models (Kohonen 1984, Personnaz er a1 1985, Kanter and Sompolinsky 1987. At present, many questions still appear to be open, concerning architectures, learning rules, storage prescriptions, etc.…”

Capacity bounds for the CDMA system and a neural network: a moderate deviations approach

“…, M, which can be written as C = ξξ + , where ξ + is the Moore-Penrose pseudoinverse of ξ. Some papers, based on not totally rigourous techniques and simulations, indicate that this rule allows a higher capacity than the classical Hebbian learning rule (see [12,13,19]). But theoretically, this rule involves calculating the inverse of a M × M matrix to get the pseudoinverse matrix.…”

“…(1) The dynamics proposed by Amari and Yanai is in fact related to the pseudo-inverse learning rule, also called projection learning rule ( [12,13,19]). The idea of this learning rule is to search a matrix C which guarantees the stability of all original patterns through the strong conditions Cξ μ = ξ μ , for all μ = 1, .…”