Representation learning for neural population activity with Neural Data Transformers

Ye, Joel; Pandarinath, Chethan

doi:10.1101/2021.01.16.426955

Cited by 14 publications

(22 citation statements)

References 35 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…When successful, representations learned from populations of neurons can provide insights into how neural circuits work to encode their inputs and drive decisions, and allow for robust and stable decoding of these correlates. Over the last decade, a number of unsupervised learning approaches have been introduced to build representations of neural population activity agnostic to specific labels or downstream decoding tasks (7; 8; 9; 10; 11; 12; 13; 14). Such methods have provided exciting new insights into the stability of neural responses (15), individual differences (11), and remapping of neural responses through learning (16).…”

Section: Introductionmentioning

confidence: 99%

Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity

Liu

Azabou

Dabagia

et al. 2021

Preprint

View full text Add to dashboard Cite

Meaningful and simplified representations of neural activity can yield insights into how and what information is being processed within a neural circuit. However, without labels, finding representations that reveal the link between the brain and behavior can be challenging. Here, we introduce a novel unsupervised approach for learning disentangled representations of neural activity called Swap-VAE. Our approach combines a generative modeling framework with an instance-specific alignment loss that tries to maximize the representational similarity between transformed views of the input (brain state). These transformed (or augmented) views are created by dropping out neurons and jittering samples in time, which intuitively should lead the network to a representation that maintains both temporal consistency and invariance to the specific neurons used to represent the neural state. Through evaluations on both synthetic data and neural recordings from hundreds of neurons in different primate brains, we show that it is possible to build representations that disentangle neural datasets along relevant latent dimensions linked to behavior.

show abstract

Section: Introductionmentioning

confidence: 99%

Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity

Liu

Azabou

Dabagia

et al. 2021

Preprint

View full text Add to dashboard Cite

show abstract

“…Conditioning of 137 neurons (i.e using 45 held-out neurons), we obtained a co-smoothing of 0.331 ± 0.001 (over 5 random seeds). For comparison, Pei et al (2021) reports 0.187 for GPFA (Yu et al, 2009), 0.225 for SLDS (Linderman et al, 2017), 0.329 for Neural Data Transformers (Ye and Pandarinath, 2021) and R 2 = 0.346 for AutoLFADS (LFADS with large scale hyperparameter optimization; Keshtkaran et al, 2021) on the same dataset.…”

Section: Experiments and Resultsmentioning

confidence: 99%

iLQR-VAE : control-based learning of input-driven dynamics with applications to neural data

Schimel

Kao

Jensen

et al. 2021

Preprint

View full text Add to dashboard Cite

Understanding how neural dynamics give rise to behaviour is one of the most fundamental questions in systems neuroscience. To achieve this, a common approach is to record neural populations in behaving animals, and model these data as emanating from a latent dynamical system whose state trajectories can then be related back to behavioural observations via some form of decoding. As recordings are typically performed in localized circuits that form only a part of the wider implicated network, it is important to simultaneously learn the local dynamics and infer any unobserved external input that might drive them. Here, we introduce iLQR-VAE, a novel control-based approach to variational inference in nonlinear dynamical systems, capable of learning both latent dynamics, initial conditions, and ongoing external inputs. As in recent deep learning approaches, our method is based on an input-driven sequential variational autoencoder (VAE). The main novelty lies in the use of the powerful iterative linear quadratic regulator algorithm (iLQR) in the recognition model. Optimization of the standard evidence lower-bound requires differentiating through iLQR solutions, which is made possible by recent advances in differentiable control. Importantly, having the recognition model implicitly defined by the generative model greatly reduces the number of free parameters and allows for flexible, high-quality inference. This makes it possible for instance to evaluate the model on a single long trial after training on smaller chunks. We demonstrate the effectiveness of iLQR-VAE on a range of synthetic systems, with autonomous as well as input-driven dynamics. We further show state-of-the-art performance on neural and behavioural recordings in non-human primates during two different reaching tasks.

show abstract

“…Through combining our approach with this nonlocal view mining strategy, we may be able to build even further invariance into our model's content space. Combining our SSL-backed approach with a sequential encoder [10] or transformer [50] is another exciting line of future research that can be used to model latent structure over longer timescales.…”

Section: Discussionmentioning

confidence: 99%

Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity

Liu,

Azabou,

Dabagia

et al. 2021

Preprint

View full text Add to dashboard Cite

Meaningful and simplified representations of neural activity can yield insights intohow and what information is being processed within a neural circuit. However, without labels, finding representations that reveal the link between the brain and behavior can be challenging. Here, we introduce a novel unsupervised approach for learning disentangled representations of neural activity called Swap-VAE. Our approach combines a generative modeling framework with an instance-specific alignment loss that tries to maximize the representational similarity between transformed views of the input (brain state). These transformed (or augmented) views are created by dropping out neurons and jittering samples in time, which intuitively should lead the network to a representation that maintains both temporal consistency and invariance to the specific neurons used to represent the neural state. Through evaluations on both synthetic data and neural recordings from hundreds of neurons in different primate brains, we show that it is possible to build representations that disentangle neural datasets along relevant latent dimensions linked to behavior.

show abstract

Representation learning for neural population activity with Neural Data Transformers

Cited by 14 publications

References 35 publications

Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity

Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity

iLQR-VAE : control-based learning of input-driven dynamics with applications to neural data

Drop, Swap, and Generate: A Self-Supervised Approach for Generating Neural Activity

Contact Info

Product

Resources

About