Hierarchical self-programming in recurrent neural networks

Self Cite

We study extremely diluted spin models of neural networks in which the connectivity evolves in time, although adiabatically slowly compared to the neurons, according to stochastic equations which on average aim to reduce frustration. The (fast) neurons and (slow) connectivity variables equilibrate separately, but at different temperatures. Our model is exactly solvable in equilibrium. We obtain phase diagrams upon making the condensed ansatz (i.e. recall of one pattern). These show that, as the connectivity temperature is lowered, the volume of the retrieval phase diverges and the fraction of mis-aligned spins is reduced. Still one always retains a region in the retrieval phase where recall states other than the one corresponding to the 'condensed' pattern are locally stable, so the associative memory character of our model is preserved.

Section: Resultsmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Slowly evolving connectivity in recurrent neural networks: I. The extreme dilution regime

Wemmenhove¹,

Skantzos²,

Coolen³

2004

Self Cite

“…We believe our method to have a number of possible advantages. It appears more direct and explicit than existing approaches, it can be generalized in a straightforward manner to situations with RSB (which could for instance be induced by super-imposed long-range bonds), and it does not rely on the limit n → 0 being taken (so that it can also be used for finite n replica calculations describing models where the disorder is not truly frozen but evolving on very large time scales, in the sense of [10][11][12][13]). …”

Section: Introductionmentioning

confidence: 99%

Diagonalization of replicated transfer matrices for disordered Ising spin systems

Nikoletopoulos¹,

Coolen²

2004

Self Cite

We present an alternative procedure for solving the eigenvalue problem of replicated transfer matrices describing disordered spin systems with (random) 1D nearest neighbour bonds and/or random fields, possibly in combination with (random) long range bonds. Our method is based on transforming the original eigenvalue problem for a 2 n × 2 n matrix (where n → 0) into an eigenvalue problem for integral operators. We first develop our formalism for the Ising chain with random bonds and fields, where we recover known results. We then apply our methods to models of spins which interact simultaneously via a onedimensional ring and via more complex long-range connectivity structures, e.g., (1 + ∞)-dimensional neural networks and 'small-world' magnets. Numerical simulations confirm our predictions satisfactorily.

“…Thus spins are always at equilibrium with respect to the dynamics of the connections. This important class of models which can be solved exactly has been first studied for fully-connected systems and Hopfield models [18,19,20,21,22,23,24], whereas, curiously, the extension of these ideas to spin systems evolving on a hierarchy of timescales reproduces the Parisi full replica symmetry breaking scheme [25].…”

Section: Introductionmentioning

confidence: 99%

Slowly evolving random graphs II: adaptive geometry in finite-connectivity Hopfield models

Wemmenhove¹,

Skantzos²

2004

We present an analytically solvable random graph model in which the connections between the nodes can evolve in time, adiabatically slowly compared to the dynamics of the nodes. We apply the formalism to finite connectivity attractor neural network (Hopfield) models and we show that due to the minimisation of the frustration effects the retrieval region of the phase diagram can be significantly enlarged. Moreover, the fraction of misaligned spins is reduced by this effect, and is smaller than in the infinite connectivity regime. The main cause of this difference is found to be the non-zero fraction of sites with vanishing local field when the connectivity is finite.