Neural Data Transformer 2: Multi-context Pretraining for Neural Spiking Activity

Ye, Joel; Collinger, Jennifer L.; Wehbe, Leila; Gaunt, Robert

doi:10.1101/2023.09.18.558113

Cited by 4 publications

(1 citation statement)

References 56 publications

(101 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Thus, behavior or neural activity preceding this segment cannot influence subsequent predictions. Integrating transformer- based approaches ( Vaswani et al, 2017 ; Jaegle et al, 2022 ) might be useful for capturing such interactions over varying timescales ( Ye and Pandarinath, 2021 ; Ye et al, 2023 ; Azabou et al, 2023 ; Antoniades et al, 2024 ).…”

Section: Limitationsmentioning

confidence: 99%

Modeling conditional distributions of neural and behavioral data with masked variational autoencoders

Schulz,

Vetter,

Gao

et al. 2024

Preprint

View full text Add to dashboard Cite

Extracting the relationship between high-dimensional recordings of neural activity and complex behavior is a ubiquitous problem in systems neuroscience. Toward this goal, encoding and decoding models attempt to infer the conditional distribution of neural activity given behavior and vice versa, while dimensionality reduction techniques aim to extract interpretable low-dimensional representations. Variational autoencoders (VAEs) are flexible deep-learning models commonly used to infer low-dimensional embeddings of neural or behavioral data. However, it is challenging for VAEs to accurately model arbitrary conditional distributions, such as those encountered in neural encoding and decoding, and even more so simultaneously. Here, we present a VAE-based approach for accurately calculating such conditional distributions. We validate our approach on a task with known ground truth and demonstrate the applicability to high-dimensional behavioral time series by retrieving the conditional distributions over masked body parts of walking flies. Finally, we probabilistically decode motor trajectories from neural population activity in a monkey reach task and query the same VAE for the encoding distribution of neural activity given behavior. Our approach provides a unifying perspective on joint dimensionality reduction and learning conditional distributions of neural and behavioral data, which will allow for scaling common analyses in neuroscience to today's high-dimensional multi-modal datasets.

show abstract

Section: Limitationsmentioning

confidence: 99%

Modeling conditional distributions of neural and behavioral data with masked variational autoencoders

Schulz,

Vetter,

Gao

et al. 2024

Preprint

View full text Add to dashboard Cite

show abstract

From monkeys to humans: observation-based EMG brain–computer interface decoders for humans with paralysis

Rizzoglio,

Altan,

et al. 2023

J. Neural Eng.

View full text Add to dashboard Cite

Intracortical brain-computer interfaces (iBCIs) aim to enable individuals with paralysis to control the movement of virtual limbs and robotic arms. Because patients’ paralysis prevents training a direct neural activity to limb movement decoder, most iBCIs rely on “observation-based” decoding in which the patient watches a moving cursor while mentally envisioning making the movement. However, this reliance on observed target motion for decoder development precludes its application to the prediction of unobservable motor output like muscle activity. Here, we ask whether recordings of muscle activity from a surrogate individual performing the same movement as the iBCI patient can be used as target for an iBCI decoder. We test two possible approaches, each using data from a human iBCI user and a monkey, both performing similar motor actions. In one approach, we trained a decoder to predict the electromyographic (EMG) activity of a monkey from neural signals recorded from a human. We then contrast this to a second approach, based on the hypothesis that the low-dimensional “latent” neural representations of motor behavior, known to be preserved across time for a given behavior, might also be preserved across individuals. We “transferred” an EMG decoder trained solely on monkey data to the human iBCI user after using Canonical Correlation Analysis to align the human latent signals to those of the monkey. We found that both direct and transfer decoding approaches allowed accurate EMG predictions between two monkeys and from a monkey to a human. Our findings suggest that these latent representations of behavior are consistent across animals and even primate species. These methods are an important initial step in the development of iBCI decoders that generate EMG predictions that could serve as signals for a biomimetic decoder controlling motion and impedance of a prosthetic arm, or even muscle force directly through Functional Electrical Stimulation.

show abstract

Unsupervised, piecewise linear decoding enables an accurate prediction of muscle activity in a multi-task brain computer interface

Ma,

Rizzoglio,

Bodkin

et al. 2024

Preprint

View full text Add to dashboard Cite

ObjectiveCreating an intracortical brain-computer interface (iBCI) capable of seamless transitions between tasks and contexts would greatly enhance user experience. However, the nonlinearity in neural activity presents challenges to computing a global iBCI decoder. We aimed to develop a method that differs from a globally optimized decoder to address this issue.ApproachWe devised an unsupervised approach that relies on the structure of a low-dimensional neural manifold to implement a piecewise linear decoder. We created a distinctive dataset in which monkeys performed a diverse set of tasks, some trained, others innate, while we recorded neural signals from the motor cortex (M1) and electromyographs (EMGs) from upper limb muscles. We used both linear and nonlinear dimensionality reduction techniques to discover neural manifolds and applied unsupervised algorithms to identify clusters within those spaces. Finally, we fit a linear decoder of EMG for each cluster. A specific decoder was activated corresponding to the cluster each new neural data point belonged to.Main resultsWe found clusters in the neural manifolds corresponding with the different tasks or task sub-phases. The performance of piecewise decoding improved as the number of clusters increased and plateaued gradually. With only two clusters it already outperformed a global linear decoder, and unexpectedly, it outperformed even a global recurrent neural network (RNN) decoder with 10-12 clusters.SignificanceThis study introduced a computationally lightweight solution for creating iBCI decoders that can function effectively across a broad range of tasks. EMG decoding is particularly challenging, as muscle activity is used, under varying contexts, to control interaction forces and limb stiffness, as well as motion. The results suggest that a piecewise linear decoder can provide a good approximation to the nonlinearity between neural activity and motor outputs, a result of our increased understanding of the structure of neural manifolds in motor cortex.

show abstract

Neural Data Transformer 2: Multi-context Pretraining for Neural Spiking Activity

Cited by 4 publications

References 56 publications

Modeling conditional distributions of neural and behavioral data with masked variational autoencoders

Modeling conditional distributions of neural and behavioral data with masked variational autoencoders

From monkeys to humans: observation-based EMG brain–computer interface decoders for humans with paralysis

Unsupervised, piecewise linear decoding enables an accurate prediction of muscle activity in a multi-task brain computer interface

Contact Info

Product

Resources

About