Deep learning approaches for neural decoding: from CNNs to LSTMs and spikes to fMRI

Abstract

Decoding behavior, perception, or cognitive state directly from neural signals has applications in brain-computer interface research as well as implications for systems neuroscience. In the last decade, deep learning has become the state-of-the-art method in many machine learning tasks ranging from speech recognition to image segmentation. The success of deep networks in other domains has led to a new wave of applications in neuroscience. In this article, we review deep learning approaches to neural decoding. We describe the architectures used for extracting useful features from neural recording modalities ranging from spikes to EEG. Furthermore, we explore how deep learning has been leveraged to predict common outputs including movement, speech, and vision, with a focus on how pretrained deep networks can be incorporated as priors for complex decoding targets like acoustic speech or images. Deep learning has been shown to be a useful tool for improving the accuracy and flexibility of neural decoding across a wide range of tasks, and we point out areas for future scientific development.

Deep learning for neural decoding in motor cortex

“…Unlike ANN which processes output at a given time using all inputs that are fed simultaneously, LSTM typically uses sequentially fed inputs to produce outputs. Thus, LSTM can flexibly integrate information over time [35]. CNN is a type of deep neural network that is mainly used for analyzing images.…”

“…CNN is a type of deep neural network that is mainly used for analyzing images. Different from ANN and LSTM, CNN has convolutional layers to extract meaningful local structures of images [35]. As brain imaging technologies (e.g.…”

“…In addition, even if preceding neural signals are not necessarily the direct command to control muscles, they can still carry nontrivial information regarding the upcoming movements [43]. Therefore, the decoding accuracy of movement trajectories may be further enhanced by including neural signals at time points preceding each movement position to decode [20,35]. To test this idea, we trained deep learning models in a second decoding scheme, time-delay decoding in which neural signals preceding each movement time point are incorporated.…”