Bayesian retrieval in associative memories with storage errors

Abstract

Abstract-It is well known that for finite-sized networks, onestep retrieval in the autoassociative Willshaw net is a suboptimal way to extract the information stored in the synapses. Iterative retrieval strategies are much better, but have hitherto only had heuristic justification. We show how they emerge naturally from considerations of probabilistic inference under conditions of noisy and partial input and a corrupted weight matrix. We start from the conditional probability distribution over possible patterns for retrieval. This contains all possible information that is available to an observer of the network and the initial input. Since this distribution is over exponentially many patterns, we use it to develop two approximate, but tractable, iterative retrieval methods. One performs maximum likelihood inference to find the single most likely pattern, using the (negative log of the) conditional probability as a Lyapunov function for retrieval. In physics terms, if storage errors are present, then the modified iterative update equations contain an additional antiferromagnetic interaction term and site dependent threshold values. The second method makes a mean field assumption to optimize a tractable estimate of the full conditional probability distribution. This leads to iterative mean field equations which can be interpreted in terms of a network of neurons with sigmoidal responses but with the same interactions and thresholds as in the maximum likelihood update equations. In the absence of storage errors, both models become very similiar to the Willshaw model, where standard retrieval is iterated using a particular form of linear threshold strategy.

A study of pattern recovery in recurrent correlation associative memories

“…We commence from the result proven in Appendix A, namely that a family of functions, not a single function, all give identical values of and, hence, identical performance for pattern recovery, subject to our model of pattern distributions. Moreover, in Appendix A we show that simplified recall performance measure (and, in fact, the overall performance of any RCAM) remains invariant under similarity transformation (24) of the excitation function . Here and are arbitrary constants.…”

“…For instance, McEleice et al [23] have shown that all of the patterns stored in the Hopfield memory are recoverable provided that and that that the target memory pattern is within a Hamming distance of the input pattern. However, other studies [24], [28], [26] have suggested a limit of based on vanishingly small levels of noise. The density of memory bits in this case is, therefore, 0.14 bits per synapse [24].…”

“…However, other studies [24], [28], [26] have suggested a limit of based on vanishingly small levels of noise. The density of memory bits in this case is, therefore, 0.14 bits per synapse [24]. Since the synaptic weights of the Hopfield memory require approximately bits of storage, the density of memory patterns is stored bits per memory bit.…”

“…However, this approach does not address the issue of the density of information contained in the memory. A more satisfactory measure is the information density [24], or the number of pattern bits that can be stored per bit of memory data. For the Hopfield memory, the storage capacity is rather poor, being typically much less than the length of the pattern-vector [4].…”

Enhancing the Delta Training Rule for a Single Layer Feedforward Heteroassociative Memory Neural Network

“…Ref. [11] showed how iterative retrieval strategies emerge naturally from considerations of probabilistic inference under conditions of noisy and partial input and a corrupted weight matrix. A description of the development of neural network models for noise reduction is given in [12].…”