Brain2Char: a deep architecture for decoding text from brain recordings

Sun, Pengfei; Anumanchipalli, Gopala K.; Chang, Edward F.

doi:10.1088/1741-2552/abc742

Cited by 56 publications

(51 citation statements)

References 33 publications

(34 reference statements)

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Second, deep-learning architectures do not allow easy introspection into how information is represented in the network, making it difficult to know what exact vocal features are being manipulated and lacking interpretability. As far as we know, these limitations have so far prevented the application of DNNs for the kind of experimental work reviewed here (although see Sun, Anumanchipalli, & Chang, 2019).…”

Section: A Prospective Note On Deep-learning Techniquesmentioning

confidence: 99%

Beyond Correlation: Acoustic Transformation Methods for the Experimental Study of Emotional Voice and Speech

et al. 2020

View full text Add to dashboard Cite

While acoustic analysis methods have become a commodity in voice emotion research, experiments that attempt not only to describe but to computationally manipulate expressive cues in emotional voice and speech have remained relatively rare. We give here a nontechnical overview of voice-transformation techniques from the audio signal-processing community that we believe are ripe for adoption in this context. We provide sound examples of what they can achieve, examples of experimental questions for which they can be used, and links to open-source implementations. We point at a number of methodological properties of these algorithms, such as being specific, parametric, exhaustive, and real-time, and describe the new possibilities that these open for the experimental study of the emotional voice.

show abstract

Section: A Prospective Note On Deep-learning Techniquesmentioning

confidence: 99%

Beyond Correlation: Acoustic Transformation Methods for the Experimental Study of Emotional Voice and Speech

et al. 2020

View full text Add to dashboard Cite

show abstract

“…Electrocorticography (ECoG) has also been explored for speech decoding, where researchers have successfully synthesized speech directly from neural signals [ 16 ]–[ 18 ]. ECoG has been shown to be effective for closed-set classification-based speech decoding [ 27 ]–[ 29 ] as well as open-set recognition of phonemes [ 15 ] or characters [ 30 ]. Despite this success, ECoG is invasive, requiring a craniotomy and surgery to implant electrodes into patients’ brains.…”

Section: Introductionmentioning

confidence: 99%

MEG Sensor Selection for Neural Speech Decoding

et al. 2020

View full text Add to dashboard Cite

Direct decoding of speech from the brain is a faster alternative to current electroencephalography (EEG) speller-based brain-computer interfaces (BCI) in providing communication assistance to locked-in patients. Magnetoencephalography (MEG) has recently shown great potential as a non-invasive neuroimaging modality for neural speech decoding, owing in part to its spatial selectivity over other high-temporal resolution devices. Standard MEG systems have a large number of cryogenically cooled channels/sensors (200 − 300) encapsulated within a fixed liquid helium dewar, precluding their use as wearable BCI devices. Fortunately, recently developed optically pumped magnetometers (OPM) do not require cryogens, and have the potential to be wearable and movable making them more suitable for BCI applications. This design is also modular allowing for customized montages to include only the sensors necessary for a particular task. As the number of sensors bears a heavy influence on the cost, size, and weight of MEG systems, minimizing the number of sensors is critical for designing practical MEG-based BCIs in the future. In this study, we sought to identify an optimal set of MEG channels to decode imagined and spoken phrases from the MEG signals. Using a forward selection algorithm with a support vector machine classifier we found that nine optimally located MEG gradiometers provided higher decoding accuracy compared to using all channels. Additionally, the forward selection algorithm achieved similar performance to dimensionality reduction using a stacked-sparse-autoencoder. Analysis of spatial dynamics of speech decoding suggested that both left and right hemisphere sensors contribute to speech decoding. Sensors approximately located near Broca's area were found to be commonly contributing among the higher-ranked sensors across all subjects. INDEX TERMS autoencoder, brain-computer interface, forward selection algorithm, magnetoencephalography, neural speech decoding, OPM, SVM

show abstract

“…For this reason, speech-to-text decoding networks often use architectures and methods like sequence-to-sequence models or the connectionist temporal classification loss [24,133], which are commonly used in machine translation or automated speech recognition applications. As such, several groups have decoded directly from neural signals to text during speech production or imagined handwriting using recurrent networks such as sequence-to-sequence models [96][97][98][99] (Fig. 3C).…”

Section: Speechmentioning

confidence: 99%

Deep learning approaches for neural decoding across architectures and recording modalities

Livezey

Glaser

2020

Briefings in Bioinformatics

View full text Add to dashboard Cite

Decoding behavior, perception or cognitive state directly from neural signals is critical for brain–computer interface research and an important tool for systems neuroscience. In the last decade, deep learning has become the state-of-the-art method in many machine learning tasks ranging from speech recognition to image segmentation. The success of deep networks in other domains has led to a new wave of applications in neuroscience. In this article, we review deep learning approaches to neural decoding. We describe the architectures used for extracting useful features from neural recording modalities ranging from spikes to functional magnetic resonance imaging. Furthermore, we explore how deep learning has been leveraged to predict common outputs including movement, speech and vision, with a focus on how pretrained deep networks can be incorporated as priors for complex decoding targets like acoustic speech or images. Deep learning has been shown to be a useful tool for improving the accuracy and flexibility of neural decoding across a wide range of tasks, and we point out areas for future scientific development.

show abstract

Brain2Char: a deep architecture for decoding text from brain recordings

Cited by 56 publications

References 33 publications

Beyond Correlation: Acoustic Transformation Methods for the Experimental Study of Emotional Voice and Speech

Beyond Correlation: Acoustic Transformation Methods for the Experimental Study of Emotional Voice and Speech

MEG Sensor Selection for Neural Speech Decoding

Deep learning approaches for neural decoding across architectures and recording modalities

Contact Info

Product

Resources

About