Opening the Black Box: Low-Dimensional Dynamics in High-Dimensional Recurrent Neural Networks

Abstract

Recurrent neural networks (RNNs) are useful tools for learning nonlinear relationships between time-varying inputs and outputs with complex temporal dependencies. Recently developed algorithms have been successful at training RNNs to perform a wide variety of tasks, but the resulting networks have been treated as black boxes: their mechanism of operation remains unknown. Here we explore the hypothesis that fixed points, both stable and unstable, and the linearized dynamics around them, can reveal crucial aspects of how RNNs implement their computations. Further, we explore the utility of linearization in areas of phase space that are not true fixed points but merely points of very slow movement. We present a simple optimization technique that is applied to trained RNNs to find the fixed and slow points of their dynamics. Linearization around these slow regions can be used to explore, or reverse-engineer, the behavior of the RNN. We describe the technique, illustrate it using simple examples, and finally showcase it on three high-dimensional RNN examples: a 3-bit flip-flop device, an input-dependent sine wave generator, and a two-point moving average. In all cases, the mechanisms of trained networks could be inferred from the sets of fixed and slow points and the linearized dynamics around them.

Context-dependent computation by recurrent dynamics in prefrontal cortex

“…We then ‘reversed engineered’ the model 33 to discover its mechanism of selective integration. The global features of the model activity are easily explained by the overall arrangement of fixed points of the dynamics 33 (Fig.…”

“…The global features of the model activity are easily explained by the overall arrangement of fixed points of the dynamics 33 (Fig. 5), which result from the synaptic connectivity learned during training.…”

(Q)SAR modelling of nanomaterial toxicity: A critical review

“…There are also others highlighting that ANN systems should not be seen as inexplicable models any more (I. I. Baskin, Palyulin, & Zefirov, 2009;Sussillo & Barak, 2013) since a number of methodologies facilitating the interpretation of model outcomes have been developed (I. Baskin, Ait, Halberstam, Palyulin, & Zefirov, 2002;Burden & Winkler, 1999;Guha, Stanton, & Jurs, 2005).…”

Low-dimensional dynamics of structured random networks

“…These vectors are restricted to the potentially low-dimensional subspace spanned by the right eigenvectors of $G_N^{\left(2\right)}$ with corresponding eigenvalues that have a real part greater than 1. Thus, although the network dynamics are chaotic, they are confined to a low-dimensional space, which has been suggested as a mechanism that could make computation in the network more robust [22]. …”