Collective Behavior of a Small-World Recurrent Neural System With Scale-Free Distribution

Abstract

This paper proposes a scale-free highly-clustered echo state network (SHESN). We designed the SHESN to include a naturally evolving state reservoir according to incremental growth rules that account for the following features:(1) short characteristic path length, (2) high clustering coefficient, (3) scale-free distribution, and (4) hierarchical and distributed architecture. This new state reservoir contains a large number of internal neurons that are sparsely interconnected in the form of domains. Each domain comprises one backbone neuron and a number of local neurons around this backbone. Such a natural and efficient recurrent neural system essentially interpolates between the completely regular Elman network and the completely random echo state network (ESN) proposed by H. Jaeger et al. We investigated the collective characteristics of the proposed complex network model. We also successfully applied it to challenging problems such as the Mackey-Glass dynamic system and the laser time series prediction.Compared to the ESN, our experimental results show that the SHESN model has a significantly enhanced echo state property and better performance in approximating highly complex nonlinear dynamics. In a word, this large-scale dynamic complex network reflects some natural characteristics of biological neural systems in many aspects such as power law, small-world property, and hierarchical architecture. It should have strong computing power, fast signal propagation speed, and coherent synchronization. Index TermsEcho state network, local preferential attachments, recurrent neural networks, scale-free, small-world, time series prediction. Manuscript

Simple Deterministically Constructed Cycle Reservoirs with Regular Jumps

“…Many other forms of reservoirs can be found in the literature (e.g. (Jones et al, 2007;Deng & Zhang, 2007;Dockendorf et al, 2009;Bush & Anderson, 2005;Ishii et al, 2004;Schmidhuber et al, 2007;Ajdari Rad et al, 2008)). However, exactly what aspects of reservoirs are responsible for their often reported superior modelling capabilities (Jaeger, 2001(Jaeger, , 2002aJaeger & Hass, 2004;Maass et al, 2004;Tong et al, 2007) is still unclear.…”

“…Our CRJ reservoirs can also be related to the work of Deng & Zhang (2007) where massive reservoirs are constructed in a randomized manner so that they exhibit smallworld and scale-free properties of complex networks. We refer to this model as the small world network reservoir (SWNR).…”

“…We refer to this model as the small world network reservoir (SWNR). We trained such SWNR architecture 8 on the laser and NARMA datasets, since for reasonable results the SWNR model needed to be of larger size, we conducted the comparative experiments with reservoirs of size 7 possibly through trained connections 8 We are thankful to the authors of (Deng & Zhang, 2007) for providing us with their code. Table 6 .…”

“…There have also been attempts to impose specialized interconnection topolo-gies on the reservoir, e.g. hierarchical reservoirs (Jaeger, 2007), small-world reservoirs (Deng & Zhang, 2007) and decoupled sub-reservoirs (Xue et al, 2007).…”

Iterative temporal learning and prediction with the sparse online echo state gaussian process

“…In particular, echo state networks (ESNs) [3] have demonstrated not only significant performance gains, but also simplified training over traditional RNN models; compared to canonical training where all weights in a network are adapted, only the mapping from a fixed neural network (the reservoir) are trained. The structure of reservoirs has a substantial impact on performance, resulting in an abundance of fresh research into optimising reservoir topology (e.g, [4], [5]). …”