Neurolinguistic and machine-learning perspectives on direct speech BCIs for restoration of naturalistic communication

Iljina, Olga; Derix, Johanna; Schirrmeister, Robin Tibor; Schulze‐Bonhage, Andreas; Auer, Péter; Aertsen, Ad; Ball, Tonio

doi:10.1080/2326263x.2017.1330611

Cited by 31 publications

(34 citation statements)

References 98 publications

Supporting

Mentioning

Contrasting

Unclassified

Order By: Relevance

“…Neurolinguistics research is providing insight into the cognitive function, phenomenology, and neurobiology of speech production in general ( Hickok, 2014 ) and imagined speech in particular ( Alderson-Day and Fernyhough, 2015 , Perrone-Bertolotti et al., 2014 ), and it is our view that these insights should be utilized within DS-BCI research. We concur with the arguments expressed by Iljina et al. (2017) that, given the complexity of speech production processes, combining research from the fields of BCI and neurolinguistics must be seen as an important approach for those seeking to capture and decode the phenomena.…”

Section: Seeking a Naturalistic Form Of Communication Through Direct-supporting

confidence: 84%

“…The importance of consistency in the way imagined speech is produced by experimental participants and the effect of providing them with a thorough understanding of what is meant by imagined speech production are additional areas for investigation that may improve the robustness of experimentation. In the following subsections, we extend the work of Iljina et al. (2017) by highlighting three key areas where BCI research can benefit from findings in the field of neurolinguistics.…”

Section: An Enhanced Role For Linguistics In Bci Researchmentioning

confidence: 97%

“…A direct-speech brain-computer interface (DS-BCI) is one that captures and decodes neural signals corresponding directly to speech production, enabling a naturalistic mode of communication ( Iljina et al., 2017 ). Such a system has the potential to transform the lives of patients with severe motor dysfunction, including pathologies such as amyotrophic lateral sclerosis resulting in locked-in syndrome.…”

Section: Seeking a Naturalistic Form Of Communication Through Direct-mentioning

confidence: 99%

See 2 more Smart Citations

Neurolinguistics Research Advancing Development of a Direct-Speech Brain-Computer Interface

2018

View full text Add to dashboard Cite

A direct-speech brain-computer interface (DS-BCI) acquires neural signals corresponding to imagined speech, then processes and decodes these signals to produce a linguistic output in the form of phonemes, words, or sentences. Recent research has shown the potential of neurolinguistics to enhance decoding approaches to imagined speech with the inclusion of semantics and phonology in experimental procedures. As neurolinguistics research findings are beginning to be incorporated within the scope of DS-BCI research, it is our view that a thorough understanding of imagined speech, and its relationship with overt speech, must be considered an integral feature of research in this field. With a focus on imagined speech, we provide a review of the most important neurolinguistics research informing the field of DS-BCI and suggest how this research may be utilized to improve current experimental protocols and decoding techniques. Our review of the literature supports a cross-disciplinary approach to DS-BCI research, in which neurolinguistics concepts and methods are utilized to aid development of a naturalistic mode of communication.

show abstract

Section: Seeking a Naturalistic Form Of Communication Through Direct-supporting

confidence: 84%

Section: An Enhanced Role For Linguistics In Bci Researchmentioning

confidence: 97%

Section: Seeking a Naturalistic Form Of Communication Through Direct-mentioning

confidence: 99%

See 1 more Smart Citation

Neurolinguistics Research Advancing Development of a Direct-Speech Brain-Computer Interface

2018

View full text Add to dashboard Cite

show abstract

“…Instead of using isolated phonemes or syllables, we collected neural data during the imagination and production of phrases (e.g., how are you? ), with the eventual goal of open-vocabulary decoding (decoding phonemes within phrases) for naturalistic communication (Iljina et al, 2017). Here, we classified whole phrases, as a starting point.…”

Section: Introductionmentioning

confidence: 99%

Decoding Imagined and Spoken Phrases From Non-invasive Neural (MEG) Signals

2020

View full text Add to dashboard Cite

Speech production is a hierarchical mechanism involving the synchronization of the brain and the oral articulators, where the intention of linguistic concepts is transformed into meaningful sounds. Individuals with locked-in syndrome (fully paralyzed but aware) lose their motor ability completely including articulation and even eyeball movement. The neural pathway may be the only option to resume a certain level of communication for these patients. Current brain-computer interfaces (BCIs) use patients' visual and attentional correlates to build communication, resulting in a slow communication rate (a few words per minute). Direct decoding of imagined speech from the neural signals (and then driving a speech synthesizer) has the potential for a higher communication rate. In this study, we investigated the decoding of five imagined and spoken phrases from single-trial, non-invasive magnetoencephalography (MEG) signals collected from eight adult subjects. Two machine learning algorithms were used. One was an artificial neural network (ANN) with statistical features as the baseline approach. The other was convolutional neural networks (CNNs) applied on the spatial, spectral and temporal features extracted from the MEG signals. Experimental results indicated the possibility to decode imagined and spoken phrases directly from neuromagnetic signals. CNNs were found to be highly effective with an average decoding accuracy of up to 93% for the imagined and 96% for the spoken phrases.

show abstract

“…which are lost once transformed into text. For these reasons, it would be desirable for spoken communication to directly synthesize an audible speech waveform from the neural recordings [15].…”

Section: Introductionmentioning

confidence: 99%

Speech Synthesis from ECoG using Densely Connected 3D Convolutional Neural Networks

Angrick

Mugler²,

Tate³

et al. 2018

Preprint

View full text Add to dashboard Cite

Objective. Direct synthesis of speech from neural signals could provide a fast and natural way of communication to people with neurological diseases. Invasively-measured brain activity (electrocorticography; ECoG) supplies the necessary temporal and spatial resolution to decode fast and complex processes such as speech production. A number of impressive advances in speech decoding using neural signals have been achieved in recent years, but the complex dynamics are still not fully understood. However, it is unlikely that simple linear models can capture the relation between neural activity and continuous spoken speech. Approach. Here we show that deep neural networks can be used to map ECoG from speech production areas onto an intermediate representation of speech (logMel spectrogram). The proposed method uses a densely connected convolutional neural network topology which is well-suited to work with the small amount of data available from each participant. Main results. In a study with six participants, we achieved correlations up to r = 0.69 between the reconstructed and original logMel spectrograms. We transfered our prediction back into an audible waveform by applying a Wavenet vocoder. The vocoder was conditioned on logMel features that harnessed a much larger, pre-existing data corpus to provide the most natural acoustic output. Significance. To the best of our knowledge, this is the first time that high-quality speech has been reconstructed from neural recordings during speech production using deep neural networks.

show abstract

Neurolinguistic and machine-learning perspectives on direct speech BCIs for restoration of naturalistic communication

Cited by 31 publications

References 98 publications

Neurolinguistics Research Advancing Development of a Direct-Speech Brain-Computer Interface

Neurolinguistics Research Advancing Development of a Direct-Speech Brain-Computer Interface

Decoding Imagined and Spoken Phrases From Non-invasive Neural (MEG) Signals

Speech Synthesis from ECoG using Densely Connected 3D Convolutional Neural Networks

Contact Info

Product

Resources

About