Decoding hindlimb kinematics from primate motor cortex using long short-term memory recurrent neural networks

Wang, Y; Truccolo, Wilson; Da, Borton

doi:10.1109/embc.2018.8512609

Cited by 9 publications

(13 citation statements)

References 7 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Deep learning methods for decoding movement have been applied to a wide range of problems. Researchers have used many input signals that have high temporal resolution, including spikes [28,[65][66][67][68][69][70], wide-band [71,72], LFP [44,49], EEG [73,74], and ECoG [53,[75][76][77]. Additionally, deep learning has been used to predict many different outputs.…”

Section: Movementmentioning

confidence: 99%

See 1 more Smart Citation

Deep learning approaches for neural decoding: from CNNs to LSTMs and spikes to fMRI

Livezey¹,

Glaser²

2020

Preprint

View full text Add to dashboard Cite

Decoding behavior, perception, or cognitive state directly from neural signals has applications in brain-computer interface research as well as implications for systems neuroscience. In the last decade, deep learning has become the state-of-the-art method in many machine learning tasks ranging from speech recognition to image segmentation. The success of deep networks in other domains has led to a new wave of applications in neuroscience. In this article, we review deep learning approaches to neural decoding. We describe the architectures used for extracting useful features from neural recording modalities ranging from spikes to EEG. Furthermore, we explore how deep learning has been leveraged to predict common outputs including movement, speech, and vision, with a focus on how pretrained deep networks can be incorporated as priors for complex decoding targets like acoustic speech or images. Deep learning has been shown to be a useful tool for improving the accuracy and flexibility of neural decoding across a wide range of tasks, and we point out areas for future scientific development.

show abstract

Section: Movementmentioning

confidence: 99%

“…Additionally, deep learning has been used to predict many different outputs. Often the output is a continuous variable, such as the position, angle, or velocity of a limb, joint, or cursor [28,44,49,53,65,66,69,70,73], or a muscles EMG [67] (Fig. 3B).…”

Section: Movementmentioning

confidence: 99%

Deep learning approaches for neural decoding: from CNNs to LSTMs and spikes to fMRI

Livezey¹,

Glaser²

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…For the linear method, most MVPA studies have used linear discriminant analysis [27], [28], general linear model [29] linear support vector machine [30], [31] and Gaussian Naive Bayes [32], [33]. For the non-linear method, nonlinear support vector machines [34], CNN [12]- [14] and LSTM [15], [36] have shown good performance in brain decoding.…”

Section: A Brain Decodingmentioning

confidence: 99%

Decoding Behavior Tasks From Brain Activity Using Deep Transfer Learning

et al. 2019

View full text Add to dashboard Cite

Recently, advances in noninvasive detection techniques have shown that it is possible to decode visual information from measurable brain activities. However, these studies typically focused on the mapping between neural activities and visual information, such as the image or video stimulus, on the individual level.Here, the common decoding models across individuals that classifying behavior tasks from brain signals were investigated. We proposed a cross-subject decoding approach using deep transfer learning (DTL) to decipher the behavior tasks from functional magnetic resonance imaging (fMRI) recording during subjects performing different tasks. We connected parts of the state-of-the-art networks pre-trained on the ImageNet dataset to our defined adaption layers to classify the behavior tasks from fMRI data. Our experiments on the Human Connectome Project (HCP) dataset showed that the proposed method achieved a higher decoding accuracy across subjects than the previous studies. We also conducted an experiment on five subsets of HCP data, which further demonstrated that our DTL approach is more effective on small dataset than the traditional methods.INDEX TERMS Neural decoding, functional magnetic resonance imaging, cross-subject, deep transfer learning.

show abstract

“…Thus, the length of the neural sequence, a hyperparameter for decoding, is usually designed manually by trial and error. In the literature, various terminologies are used for the length of a neural sequence, including number of timesteps [ 8 , 9 ], number of taps [ 6 , 10 ], length of the sliding window [ 4 ] and time window size [ 7 , 11 ]. We refer to the length of a neural sequence for decoding as the number of timesteps for consistency with most neural decoders and considering its use in neuroscience [ 12 ].…”

Section: Introductionmentioning

confidence: 99%

“…Recent advances in recurrent neural networks (RNNs) have led to improved neural decoder designs [ 9 , 13 ] and real-time BCI systems [ 14 ]. RNN-based neural decoders learn the neural response dynamics from the neural activity in both the previous and current timesteps (time bins) [ 6 ].…”

Section: Introductionmentioning

confidence: 99%

Selection of Essential Neural Activity Timesteps for Intracortical Brain–Computer Interface Based on Recurrent Neural Network

Yang

Huang

et al. 2021

Sensors

View full text Add to dashboard Cite

Intracortical brain–computer interfaces (iBCIs) translate neural activity into control commands, thereby allowing paralyzed persons to control devices via their brain signals. Recurrent neural networks (RNNs) are widely used as neural decoders because they can learn neural response dynamics from continuous neural activity. Nevertheless, excessively long or short input neural activity for an RNN may decrease its decoding performance. Based on the temporal attention module exploiting relations in features over time, we propose a temporal attention-aware timestep selection (TTS) method that improves the interpretability of the salience of each timestep in an input neural activity. Furthermore, TTS determines the appropriate input neural activity length for accurate neural decoding. Experimental results show that the proposed TTS efficiently selects 28 essential timesteps for RNN-based neural decoders, outperforming state-of-the-art neural decoders on two nonhuman primate datasets (R2=0.76±0.05 for monkey Indy and CC=0.91±0.01 for monkey N). In addition, it reduces the computation time for offline training (reducing 5–12%) and online prediction (reducing 16–18%). When visualizing the attention mechanism in TTS, the preparatory neural activity is consecutively highlighted during arm movement, and the most recent neural activity is highlighted during the resting state in nonhuman primates. Selecting only a few essential timesteps for an RNN-based neural decoder provides sufficient decoding performance and requires only a short computation time.

show abstract

Decoding hindlimb kinematics from primate motor cortex using long short-term memory recurrent neural networks

Cited by 9 publications

References 7 publications

Deep learning approaches for neural decoding: from CNNs to LSTMs and spikes to fMRI

Deep learning approaches for neural decoding: from CNNs to LSTMs and spikes to fMRI

Decoding Behavior Tasks From Brain Activity Using Deep Transfer Learning

Selection of Essential Neural Activity Timesteps for Intracortical Brain–Computer Interface Based on Recurrent Neural Network

Contact Info

Product

Resources

About