Deep learning approaches for neural decoding across architectures and recording modalities

Abstract

Decoding behavior, perception or cognitive state directly from neural signals is critical for brain–computer interface research and an important tool for systems neuroscience. In the last decade, deep learning has become the state-of-the-art method in many machine learning tasks ranging from speech recognition to image segmentation. The success of deep networks in other domains has led to a new wave of applications in neuroscience. In this article, we review deep learning approaches to neural decoding. We describe the architectures used for extracting useful features from neural recording modalities ranging from spikes to functional magnetic resonance imaging. Furthermore, we explore how deep learning has been leveraged to predict common outputs including movement, speech and vision, with a focus on how pretrained deep networks can be incorporated as priors for complex decoding targets like acoustic speech or images. Deep learning has been shown to be a useful tool for improving the accuracy and flexibility of neural decoding across a wide range of tasks, and we point out areas for future scientific development.

A wearable platform for closed-loop stimulation and recording of single-neuron and local field potential activity in freely-moving humans

“…Given the increasing benefit of using machine learning approaches [47][48][49] in neuroscience studies, the Neuro-stack could be useful for validating decoding models and testing novel closedloop stimulation therapies (e.g., to improve memory in patients with severe memory impairments).…”

Incorporating structural knowledge into unsupervised deep learning for two-photon imaging data

“…Deep learning has been shown to be a useful tool for analyzing biomedical imaging data across a wide range of tasks. Specifically, such approaches have been developed for neural decoding [9] and are increasingly becoming a part of analysis workflows for live cell imaging data, such as for two-photon imaging [14, 17, 18]. Here, automatic extraction and modeling of neural activity from individual cells over time and in large cell populations is a crucial step towards a better understanding of cellular activity that can ultimately be linked to phenotype data to facilitate biological insight.…”

“…Here, two-photon microscopy is used to capture images of a neuronal population that expresses a fluorescent calcium indicator and allows to visualize the increase in intracellular Ca2+-concentration accompanying neurons’ spiking activity [5]. More generally, ‘neural decoding’ refers to techniques that use brain signals to make predictions about behaviour, perception, or cognitive state and are becoming increasingly important for neuroscientific research [6–9]. For example, calcium imaging techniques enable optical measurement of large neural populations at a high spatio-temporal resolution and thus facilitate insights into neural activity [5, 10, 11].…”

“…To realize their full potential, computational methods are needed to derive a model of neural activity from the indirect optical measurement of the Ca2+ indicator. Here, to analyze the resulting video data, the typical two-step analysis framework of cell population imaging is employed, where cells are identified through segmentation in a first step, and the signals of interest, e.g., temporal fluorescence traces or firing rates, are identified subsequently [9, 12–14]. Current approaches to derive an explicit generative model typically rely on simplifying assumptions of, e.g., a linear combination of additive signals, that allow to directly incorporate known structure, but limit the flexibility and versatility of the obtained data model [15, 16].…”

“…Additionally, these approaches often only provide one of several components in the entire data analysis workflow [14], where prior steps, such as motion correction, can considerably affect performance [20]. Yet, results showing that temporal information can improve segmentation [17] indicate that it might be beneficial to consider both steps simultaneously and perform modeling directly based on the temporal sequence of images without explicitly extracting traces [9]. Such an integrated modeling strategy can help to build a flexible and versatile model that is not limited to a specific task.…”