Training noise adaptation in attractor neural networks

Wong, K. Y. Michael; Sherrington, David C.

doi:10.1088/0305-4470/23/4/009

Cited by 31 publications

(20 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…5 Fig. 6(a), training noises disrupt the stored patterns and reduce the attractor overlap, as found previously [14]. On the other hand, the storage is sufficiently low that the boundary overlap is always zero.…”

Section: Introductionsupporting

confidence: 67%

“…This means that the network is always in the wide retrieval region. Assuming that training noises enhance network associativity as found previously [14], the MSN, corresponding to a training overlap of m, =1, is supposed to have the least associativity. Since even the MSN has wide retrieval basins for a below 0.42 [31], this scenario of wide retrieval basins for all training overlaps extends up to the storage level 0.42.…”

Section: Introductionmentioning

confidence: 96%

“…In the context of attractor networks, we have shown that training noises improve the associativity of the networks, i.e. , they widen the basins of attraction of the stored patterns [14]. Furthermore, it has also been shown that training noises enhance the robustness of retrieval against thermal noise in the retrieving dynamics [9,19] and network damage [20].…”

Section: Introductionmentioning

confidence: 98%

“…We have studied the performance optimization for noisy training and retrieving conditions [9,14]. Noises in neural networks may be present in the training data, the data to be retrieved, or the retrieval dynamics.…”

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations

Neural networks optimally trained with noisy data

Wong

Sherrington

1993

Phys. Rev. E

View full text Add to dashboard Cite

We study the retrieval behaviors of neural networks which are trained to optimize their performance for an ensemble of noisy example patterns. In particular, we consider (1) the performance overlap, which rejects the performance of the network in an operating condition identical to the training condition; (2) the storage overlap, which reAects the ability of the network to merely memorize the stored information; (3) the attractor overlap, which rejects the precision of retrieval for dilute feedback networks; and (4) the boundary overlap, which defines the boundary of the basin of attraction, and hence the associative ability for dilute feedback networks. We find that for suKciently low training noise, the network optimizes its overall perforance by sacrificing the individual performance of a minority of patterns, resulting in a two-band distribution of the aligning fields. For a narrow range of storage level, the network loses and then regains its retrieval capability when the training noise level increases, and we interpret that this reentrant retrieval behavior is related to competing tendencies in structuring the basins of attraction for the stored patterns. Reentrant behavior is also observed in the space of synaptic interactions, in which the replica symmetric solution of the optimal network destabilizes and then restabilizes when the training noise level increases. We summarize these observations by picturing training noises as an instrument for widening the basins of attractions of the stored patterns at the expense of reducing the precision of retrieval.

show abstract

Section: Introductionsupporting

confidence: 67%

Section: Introductionmentioning

confidence: 96%

Section: Introductionmentioning

confidence: 98%

Section: Introductionmentioning

confidence: 99%

See 2 more Smart Citations

Neural networks optimally trained with noisy data

Wong

Sherrington

1993

Phys. Rev. E

View full text Add to dashboard Cite

show abstract

“…When the shortage cost is comparable to the transportation cost, the total cost may be optimized either by saving the transportation cost feeding a poor node while sacrificing the satisfaction of the node, or by saving the shortage cost while spending more on the transportation cost. The picture in reminiscent of the learning of noisy examples in perceptrons, where the field distribution of the examples consist of the bands, corresponding to the learned and sacrificed examples respectively [15,16,17,18]. As a result, frustration arises from competition for resources among connected nodes.…”

Section: Introductionmentioning

confidence: 99%

Optimal location of sources in transportation networks

Yeung¹,

Wong²

2010

J. Stat. Mech.

View full text Add to dashboard Cite

Abstract. We consider the problem of optimizing the locations of source nodes in transportation networks. A reduction of the fraction of surplus nodes induces a glassy transition. In contrast to most constraint satisfaction problems involving discrete variables, our problem involves continuous variables which lead to cavity fields in the form of functions. The one-step replica symmetry breaking (1RSB) solution involves solving a stable distribution of functionals, which is in general infeasible. In this paper, we obtain small closed sets of functional cavity fields and demonstrate how functional recursions are converted to simple recursions of probabilities, which make the 1RSB solution feasible. The physical results in the replica symmetric (RS) and the 1RSB frameworks are thus derived and the stability of the RS and 1RSB solutions are examined.

show abstract

Statistical Mechanics of Neural Networks

Garrido¹

1990

Lecture Notes in Physics

View full text Add to dashboard Cite

We investigate five different problems in the field of the statistical mechanics of neural networks.The first three problems involve attractor neural networks that optimise particular cost functions for storage of static memories as attractors of the neural dynamics. We study the effects of replica symmetry breaking (RSB) and attempt to find algorithms that will produce the optimal network if error-free storage is impossible.For the Gardner-Derrida network we show that full RSB is necessary for an exact solution everywhere above saturation. We also show that, no matter what the cost function that is optimised, if the distribution of stabilities has a gap then the Parisi replica ansatz that has been made is unstable.For the noise-optimal network we find a continuous transition to replica symmetry breaking at the AT line, in line with previous studies of RSB for different networks. The change to RSBl improves the agreement between "experimental" and theoretical calculations of the local stability distribution p(X] significantly. The effect on observables is smaller.We show that if the network is presented with a training set which has been generated from a set of prototypes by some noisy rule, but neither the noise level nor the prototypes are known, then the perceptron algorithm is the best initial choice to produce a network that will generalise well. If additional information is available more sophisticated algorithms will be faster and give a smaller generalisation error.The remaining problems deal with attractor neural networks with separable interaction matrices which can be used (under parallel dynamics) to store sequences of patterns without the need for time delays. We look at the effects of correlations on a singlesequence network, and numerically investigate the storage capacity of a network storing an extensive number of patterns in such sequences.When correlations are implemented along with a term in the interaction matrix designed to suppress some of the effects of those correlations, the competition between the two produces a rich range of behaviour. Contrary to expectations, increasing the correlations and the operating temperature proves capable of improving the sequenceprocessing behaviour of the network.Finally, we demonstrate that a network storing a large number of sequences of patterns using a Hebb-like rule can store approximately twice as many patterns as the network trained with the Hebb rule to store individual patterns.

show abstract

Training noise adaptation in attractor neural networks

Cited by 31 publications

References 14 publications

Neural networks optimally trained with noisy data

Neural networks optimally trained with noisy data

Optimal location of sources in transportation networks

Statistical Mechanics of Neural Networks

Contact Info

Product

Resources

About