Optimal Sequence Memory in Driven Random Networks

Abstract

Autonomous randomly coupled neural networks display a transition to chaos at a critical coupling strength. We here investigate the effect of a time-varying input on the onset of chaos and the resulting consequences for information processing. Dynamic mean-field theory yields the statistics of the activity, the maximum Lyapunov exponent, and the memory capacity of the network. We find an exact condition that determines the transition from stable to chaotic dynamics and the sequential memory capacity in closed form. The input suppresses chaos by a dynamic mechanism, shifting the transition to significantly larger coupling strengths than predicted by local stability analysis. Beyond linear stability, a regime of coexistent locally expansive, but non-chaotic dynamics emerges that optimizes the capacity of the network to store sequential input.

Optimal short-term memory before the edge of chaos in driven random recurrent networks

“…An online supervised learning algorithm for RNNs proposed by Sussillo and Abbott [33] exhibits its best performance when their autonomous dynamics is adjusted to the chaotic region not far from the critical point where chaotic dynamics can be suppressed by input signals. Schuecker et al [23] showed that the network memory capacity for continuous-time nonlinear RNNs peaks in the ordered regime with g 2 > 1, which is consistent to our result. They argued that the dynamic suppression of chaos (DSC), which results from the fact that the onset of local instability precedes that of asymptotic instability, contributes to optimal information processing.…”

“…In ESNs, the shift of the critical g 2 * toward the chaotic regime induced by input signals is solely due to a mechanism called the static suppression of chaos (SSC), which increases the frequency with which an orbit visits the contracting region of the phase space. Unlike SSC, DSC is conjectured to occur based on fast switching among different unstable directions caused by input signals [23]. However, ESNs with leaky neurons [34] are expected to exhibit DSC [23].…”

“…The memory capacity of an ESN is defined as the quality of the optimal linear estimator of the past input s(t − n) using the present state of neurons x i (t). Following previous work [23,25], we assume a sparse readout, namely, there are K = O(1) readout neurons 1 ≤ i ≤ K and consider linear readoutŝ(t) = K i=1 v i x i (t). Given time-delay n (n = 1, 2, .…”

“…However, the memory capacity of nonlinear RNNs is difficult to study by analytical methods [22]. Recently, Schuecker et al [23] successfully derived an analytical expression for memory capacity for continuous-time nonlinear RNNs [24] in which each neuron is driven by independent input signals following a white-noise Gaussian process. Toyoizumi and Abbott [25] analytically calculated the signal-to-noise ratio, which is equivalent to the inverse of memory capacity at the limit of zero input strength, for discrete-time nonlinear RNNs driven by a common time-varying input signal.…”

On the number of limit cycles in asymmetric neural networks

“…Other transitions obtained with this model regard the transient time spent to reach the attractor from random inputs [13][14][15] and the size of the basins of attraction of the existing limit cycles [14]. Adding noise [16,17], dilution [18], a gain function [19][20][21] or self-interaction [22] may further expand the array of possible transitions in the system. In particular, adding dilution to fully asymmetric networks results in an increased number of attractors, and therefore to an increased storage capacity [23].…”