A complete self-control mechanism is proposed in the dynamics of neural networks through the introduction of a time-dependent threshold, determined in function of both the noise and the pattern activity in the network. Especially for sparsely coded models this mechanism is shown to considerably improve the storage capacity, the basins of attraction and the mutual information content of the network.PACS numbers: 87.10+e, 64.60CnSparsely coded models have attracted a lot of attention in the development of neural networks, both from the device oriented and biologically oriented point of view [1]- [2]. It is well-known that they have a large storage capacity, which behaves as 1/(a ln a) for a small where a is the pattern activity. However, it is clear that the basins of attraction, e.g., should not become too small because then sparse coding is, in fact, useless.In this context the necessity of an activity control system has been emphasized, which tries to keep the activity of the network in the retrieval process the same as the one for the memorized patterns [3]-[5]. This has led to several discussions imposing external constraints on the dynamics (see the references in [2]). Clearly, the enforcement of such a constraint at every time step destroys part of the autonomous functioning of the network.An important question is then whether the capacity of storage and retrieval with non-negligible basins of attraction can be improved and even be optimized without imposing these external constraints, keeping at the same time the simplicity of the architecture of the network.In this Letter we answer this question by proposing, as far as we are aware for the first time, a complete self-control mechanism in the dynamics of neural networks. This is done through the introduction of a time-dependent threshold in the transfer function. This threshold is chosen as a function of the noise in the system and the pattern activity, and adapts itself in the course of the time evolution. The difference with existing results in the literature [2] precisely lies in this adaptivity property. This immediately solves, e.g., the difficult problem of finding the mostly narrow interval for an optimal threshold such that the basins of attraction of the memorized patterns do not shrink to zero.We have worked out the practical case of sparsely coded models. We find that the storage capacity, the basins of attraction as well as the mutual information content are improved. These results are shown to be valid also for not so sparse models. Indeed, a similar selfcontrol mechanism should even work in more complicated architectures, e.g., layered and fully connected ones. Furthermore, this idea of self-control might be relevant for dynamical systems in general, when trying to improve the basins of attraction and the convergence times. 0.0 0.2 0.4 0.6 0.8 θ 0.0 0.1 0.2 i α=30 α=20 α=10 0.0 0.1 0.2 i α=0.8 α=0.6 α=0.4 a=0.1 a=10 −3 FIG. 1. The information i as a function of θ without self-control for a = 0.1 (top) and a = 0.001 (bottom) for several values ...

No abstract

No abstract

A study of the time evolution and a stability analysis of the phases in the extremely diluted Blume-EmeryGriffiths neural network model are shown to yield new phase diagrams in which fluctuation retrieval may drive pattern retrieval. It is shown that saddle-point solutions associated with fluctuation overlaps slow down the flow of the network states towards the retrieval fixed points. A comparison of the performance with other three-state networks is also presented. A novel idea suggested recently in the theory of attractor neural networks is to use information theory to infer the learning rule of an optimally performing three-state network ͓1͔. Optimal means that although the network might start initially far from the embedded pattern, i.e., having a vanishingly small initial mutual information, it is still able to retrieve it. The study of this mutual information leads to Blume-Emery-Griffiths-type ͑BEG͒ network models with Hebbian-like learning rules ͓1,2͔. Its structure also reveals that the retrieval overlap and fluctuation overlap are the relevant order parameters in order to study the network performance.It has been argued in Ref.͓1͔ that in an extremely diluted architecture of the BEG-type new states associated with the fluctuation overlap, the quadrupolar states Q appear for all values of the synaptic noise ͑temperature T). However, neither the stability of these states nor their time evolution have been discussed in detail. These are precisely the subjects of this Brief Report. In particular, we find that due to the presence of long transients in the dynamic evolution of the network, we need a finite activity dependent threshold in order to stabilize these Q states and, hence, part of the phase diagrams are altered in a substantial way. Moreover, we clarify the explicit role of the fluctuation overlap in enhancing the retrieval performance of the network compared with other three-state networks. This study further allows us to advocate the use of these Q states as new information carriers in practical applications, e.g., in pattern recognition where, looking at a black and white picture on a gray background, such a state would tell us the exact location of the picture with respect to the background without finding the details of the picture itself. Whether these states could also model such retrieval focusing problems discussed in the framework of cognitive neuroscience ͑see, e.g., Ref. ͓3͔͒ is an interesting thought.Consider a three-state network with symmetrically distributed neuron states i,t ϭ0,Ϯ1 on sites iϭ1, . . . ,N, at time step t, where i,t ϭϮ1 denote the active states. A set of p ternary patterns, ͕ i ϭ0,Ϯ1͖, ϭ1, . . . ,p, where i ϭϮ1 are the active ones, assumed to be independent random variables following the probability distributionare stored in the network. Hence, the mean ͗ i ͘ϭ0 and the variance aϭ͗ ( i ) 2 ͘ is the activity of the patterns.At each time step we regard the patterns as the inputs and the neuron states as the outputs of the network. Then we can consider the mutual inform...