Transient dynamics of sparsely connected Hopfield neural networks with arbitrary degree distributions

Abstract

Using probabilistic approach, the transient dynamics of sparsely connected Hopfield neural networks is studied for arbitrary degree distributions. A recursive scheme is developed to determine the time evolution of overlap parameters. As illustrative examples, the explicit calculations of dynamics for networks with binomial, power-law, and uniform degree distribution are performed. The results are good agreement with the extensive numerical simulations. It indicates that with the same average degree, there is a gradual improvement of network performance with increasing sharpness of its degree distribution, and the most efficient degree distribution for global storage of patterns is the delta function.As a tractable toy model of associative memories and can also be viewed as an extension of the Ising model, Hopfield neural networks [1] received lots of attention in the recent two decades. Equilibrium properties of fully connected Hopfield neural networks have been well studied using spin-glass theory, especially the replica method [2,3]. Their dynamics is also studied using the generating functional method [4] and signal-to-noise analysis [5][6][7].Given the huge number of neurons, there is only a small number of interconnections in the human brain cortex (∼10 11 neurons and ∼10 14 synapses). In order to simulate a biological genuine model rather than the fully connected networks, various random diluted models were studied, including extremely diluted model [8,9], finite diluted model [10,11], and finite connection model [12,13]. But neural connectivity is suggested to be far more complex than a fully random graph, e.g. the networks of c.elegans and cat's cortical neural were reported to be small-world and scale-free, respectively [14,15]. To go one step closer to a more biologically realistic model, many numerical studies are carried out, focusing on how the topology, the degree distribution, and clustering coefficient of a network topology affect the computational performance of the Hopfield model [16][17][18][19]. With the same average connection, a random network was reported to be more efficient for storage and retrieval of patterns than either a small-world network or regular network [17]. Torres et al. reported that the capacity of storage is higher for a neural network with

Signal-to-noise analysis of Hopfield neural networks with a formulation of the dynamics in terms of transition probabilities

“…In Refs. [27,28], models with asymmetric synapses and infinite connectivity are studied using SNA. The GFA method is applied in Refs.…”

“…Our method could be applied to extend the results in Refs. [27,28] to models with symmetric synapses. The path probabilities have an important role in the solutions of models with finite connectivity.…”

Nonbacktracking operator for the Ising model and its applications in systems with multiple states

“…A large amount of studies have been devoted to trying to increase capacity and the retrieval abilities of neural networks by optimizing the learning rule or by optimizing the topology of the network [23][24][25][26][27][28][29], In the following text of the paper we are going to discuss optimizing the topology of the network. We know that a fully connected Hopfield network can store a number of patterns proportional to the number ol neurons, but the fully connected topology is not biologicaly realistic.…”

Non-backtracking operator for Ising model and its application in attractor neural networks

“…A large amount of studies have been devoted to increase capacity and the retrieval abilities of neural networks by optimizing the learning rule or by optimizing the topology of the network [30][31][32][33][34][35][36]. In the following text of the paper we are going to discuss more on optimizing the topology of the network.…”