Hierarchical neural networks perform both serial and parallel processing

Abstract

In this work we study a Hebbian neural network, where neurons are arranged according to a hierarchical architecture such that their couplings scale with their reciprocal distance. As a full statistical mechanics solution is not yet available, after a streamlined introduction to the state of the art via that route, the problem is consistently approached through signal-to-noise technique and extensive numerical simulations. Focusing on the low-storage regime, where the amount of stored patterns grows at most logarithmical with the system size, we prove that these non-mean-field Hopfield-like networks display a richer phase diagram than their classical counterparts. In particular, these networks are able to perform serial processing (i.e. retrieve one pattern at a time through a complete rearrangement of the whole ensemble of neurons) as well as parallel processing (i.e. retrieve several patterns simultaneously, delegating the management of different patterns to diverse communities that build network). The tune between the two regimes is given by the rate of the coupling decay and by the level of noise affecting the system. The price to pay for those remarkable capabilities lies in a network's capacity smaller than the mean field counterpart, thus yielding a new budget principle: the wider the multitasking capabilities, the lower the network load and vice versa. This may have important implications in our understanding of biological complexity.

Real-space renormalization group for spectral properties of hierarchical networks

“…Here, no O(λ)-correction was needed, since the denominator of Eq. (22) is already finite at λ = 0. In contrast, the numerator cancels at λ = 0 and yields instead…”

Topological properties of hierarchical networks

“…that is, in the thermodynamic limit, the weighted degree has a logarithmical divergence with N (we recall that N = 2 K ); coherently, the case σ = 1/2 is excluded from the statistical-mechanics investigations [2,3]. The last part of this section is devoted to the study of the network modularity and clustering.…”

“…Therefore, despite the network we are considering is fully connected, when noise is present weaker weights, with J ij < T , basically do not play any longer, as if they were missing [7]. Since in the statistical mechanical analysis the noise level can be tuned arbitrarily [2,3], it is crucial to understand how the overall network connection and clustering are accordingly modified.…”

Non-convex Multi-species Hopfield Models

“…For instance, by taking P finite (or, still, sublinear with respect to N ) in the thermodynamic limit N → ∞, one has retrieval capabilities as long as T < 1. A modification of the Hebb rule results in a deformation of the basins of attractions: this can be done for example overlaying Hebb to another interaction structure as a diluted [45,46] or a hierarchical [47,48,49,50] structure. In order to describe the overall state of the system, one introduces the macroscopic observable m, also called Mattis magnetization, that is a vector of length P , whose µ-th component represents the overlap between the spin configuration and the µ-th pattern:…”