Driving reservoir models with oscillations: a solution to the extreme structural sensitivity of chaotic networks

Abstract

A large body of experimental and theoretical work on neural coding suggests that the information stored in brain circuits is represented by time-varying patterns of neural activity. Reservoir computing, where the activity of a recurrently connected pool of neurons is read by one or more units that provide an output response, successfully exploits this type of neural activity. However, the question of system robustness to small structural perturbations, such as failing neurons and synapses, has been largely overlooked. This contrasts with well-studied dynamical perturbations that lead to divergent network activity in the presence of chaos, as is the case for many reservoir networks. Here, we distinguish between two types of structural network perturbations, namely local (e.g., individual synaptic or neuronal failure) and global (e.g., network-wide fluctuations). Surprisingly, we show that while global perturbations have a limited impact on the ability of reservoir models to perform various tasks, local perturbations can produce drastic effects. To address this limitation, we introduce a new architecture where the reservoir is driven by a layer of oscillators that generate stable and repeatable trajectories. This model outperforms previous implementations while being resistant to relatively large local and global perturbations. This finding has implications for the design of reservoir models that capture the capacity of brain circuits to perform cognitively and behaviorally relevant tasks while remaining robust to various forms of perturbations. Further, our work proposes a novel role for neuronal oscillations found in cortical circuits, where they may serve as a collection of inputs from which a network can robustly generate complex dynamics and implement rich computations.

Learning long temporal sequences in spiking networks by multiplexing neural oscillations

“…The combination of N inp input oscillators will generate a sequence of unique N inp -dimensional vectors where the sequence lasts as long as the least common multiple of the inputs’ individual periods (15). For instance, two sine waves with periods of 200 ms and 250 ms would create a multi-periodic input with a period lasting 1,000 ms.…”

“…That is because RC models are either restricted to learning periodic functions, or require an aperiodic input to generate an aperiodic output, thus leaving the neural origins of aperiodic activity unresolved (5). A solution to this problem is to train the recurrent connections of the reservoir to stabilize innate patterns of activity (12), but this approach is more computationally expensive and is sensitive to structural perturbations (15).…”

“…Further, no feedback from the output units to the reservoir was required during training (11, 16). Thus, multiplexing a set of oscillators into a reservoir network provides an efficient and neurophysiologically grounded means of controlling a recurrent circuit (15). Analogous mechanisms have been hypothesized in other contexts including grid cell representations (17) and interval timing (18, 19).…”

Interpreting Recurrent Neural Networks Behaviour via Excitable Network Attractors

“…On the other hand, training of the recurrent layer in ESNs is typically realised by perturbing a randomly-initialised reservoir with a low-rank, deterministic matrix. This is conventionally accomplished by feeding back the ESN output to the recurrent layer [27,48,49,59,61] or, as Mastrogiuseppe and Ostojic [35] recently proposed, by designing the reservoir directly as W r = X + D, where X is a random matrix and D is a deterministic, low-rank matrix encoding the task of interest. (5).…”

Toward Arbitrary Control of Lattice Interactions in Nonequilibrium Condensates

“…In addition to Hamiltonian optimization and simulations of oscillators behavior, various nonlinear dynamical systems, including electronic, photonic, spintronic, mechanical, and biological systems, have been recently employed as potential reservoirs for reservoir computing (RC) (see ref. [] and references therein).…”