Probing the Relationship Between Latent Linear Dynamical Systems and Low-Rank Recurrent Neural Network Models

Abstract

A large body of work has suggested that neural populations exhibit low-dimensional dynamics during behavior. However, there are a variety of different approaches for modeling low-dimensional neural population activity. One approach involves latent linear dynamical system (LDS) models, in which population activity is described by a projection of low-dimensional latent variables with linear dynamics. A second approach involves low-rank recurrent neural networks (RNNs), in which population activity arises directly from a low-dimensional projection of past activity. Although these two modeling approaches have strong similarities, they arise in different contexts and tend to have different domains of application. Here we examine the precise relationship between latent LDS models and linear low-rank RNNs. When can one model class be converted to the other, and vice versa? We show that latent LDS models can only be converted to RNNs in specific limit cases, due to the non-Markovian property of latent LDS models. Conversely, we show that linear RNNs can be mapped onto LDS models, with latent dimensionality at most twice the rank of the RNN. A surprising consequence of our results is that a partially observed RNN is better represented by an LDS model than by an RNN consisting of only observed units.

Relating local connectivity and global dynamics in recurrent excitatory-inhibitory networks

“…Specifically, the low-rank approximation shows that the different heterogeneity between excitatory and inhibitory populations depends only on the block-structured variances, i. e. , q=E,I α p g 2 pq /λ 2 0 , and independent of the historical population activity. The full-rank dynamics, on the other hand, shows that the different heterogeneity depends on the local relationship between block structured variance and the structure of historical population activity [68]. For a simplified network example, where the locally defined connectivity has homogeneous random parameters g pq = g, p = E, I, the variance of the rank-one perturbation eigenvector σ 2 m p is thereby the same for both excitatory and inhibitory populations.…”

“…Specifically, the low-rank approximation shows that the different heterogeneity between excitatory and inhibitory populations depends only on the block-structured variances, i. e., , and independent of the historical population activity. The full-rank dynamics, on the other hand, shows that the different heterogeneity depends on the local relationship between block structured variance and the structure of historical population activity [68].…”

Relating local connectivity and global dynamics in recurrent excitatory-inhibitory networks

“…A key insight from our study is a general relationship between reciprocal motifs in locally-defined connectivity and overlaps among connectivity vectors in low-rank networks, which hasn’t been investigated in the previous works of the low-rank model [ 30 , 31 , 33 , 79 ]. Indeed, we have shown that correlations between reciprocal synaptic weights generate overlaps beyond the mean in the corresponding low-rank approximation ( Eq (95) ).…”

Geometry of population activity in spiking networks with low-rank structure

“…In rate networks with a low-rank connectivity matrix, the dynamics of the activations x ( t ) = { x i ( t )} i =1… N are explicitly confined to a low-dimensional subspace of state space [Beiran et al, 2021a, Dubreuil et al, 2022, Valente et al, 2022b], meaning that projections of x ( t ) are non-zero only on vectors w belonging to this subspace. Here we first reproduce the derivation of the geometry of the activations x ( t ) [Beiran et al, 2021a, Dubreuil et al, 2022].…”

“…A particularly fruitful approach has been to interpret the emerging computations in terms of the geometry of dynamics in the state space of joint activity of all neurons [Sussillo and Barak, 2013, Mante et al, 2013, Vyas et al, 2020, Chung and Abbott, 2021], as commonly done with experimental data [Churchland and Shenoy, 2007, Buonomano and Maass, 2009, Cunningham and Yu, 2014, Gallego et al, 2017, Saxena and Cunningham, 2019, Jazayeri and Ostojic, 2021]. In particular, in a large class of rate networks in which the connectivity contains a low-rank structure [Hopfield, 1982, Eliasmith and Anderson, 2004, Sussillo and Abbott, 2009, Boerlin et al, 2013b, Ahmadian et al, 2015, Pereira and Brunel, 2018, Landau and Sompolinsky, 2018, Beiran et al, 2021b, Landau and Sompolinsky, 2021, Schuessler et al, 2020b, Kadmon et al, 2020, Logiaco et al, 2021, Valente et al, 2022a], the geometry of activity and the resulting computations can be analytically predicted from the structure of connectivity [Mastrogiuseppe and Ostojic, 2018, Schuessler et al, 2020a, Beiran et al, 2021a, Dubreuil et al, 2022]. A comparable mechanistic picture has so far been missing in spiking networks.…”