Neurolinguistics Research Advancing Development of a Direct-Speech Brain-Computer Interface

Abstract

A direct-speech brain-computer interface (DS-BCI) acquires neural signals corresponding to imagined speech, then processes and decodes these signals to produce a linguistic output in the form of phonemes, words, or sentences. Recent research has shown the potential of neurolinguistics to enhance decoding approaches to imagined speech with the inclusion of semantics and phonology in experimental procedures. As neurolinguistics research findings are beginning to be incorporated within the scope of DS-BCI research, it is our view that a thorough understanding of imagined speech, and its relationship with overt speech, must be considered an integral feature of research in this field. With a focus on imagined speech, we provide a review of the most important neurolinguistics research informing the field of DS-BCI and suggest how this research may be utilized to improve current experimental protocols and decoding techniques. Our review of the literature supports a cross-disciplinary approach to DS-BCI research, in which neurolinguistics concepts and methods are utilized to aid development of a naturalistic mode of communication.

Decoding Imagined and Spoken Phrases From Non-invasive Neural (MEG) Signals

“…This limited 1 s duration attempted to ensure the behavioral compliance of the subject to imagine only that phrase. It is extremely difficult to verify whether the subjects actually performed the task of imagination in the traditional setting of imagined speech data acquisition (Cooney et al, 2018). Hence, we collected both imagined and overt speech in a single trial with a limited duration for the imagination segment for behavioral control.…”

“…Nevertheless, our paradigm may have a combination of both preparation and imagination within the 1 s duration. Based on the extant literature, phonological representation is shown to be activated during orthographical language processing and the preparation (pre-speech) (Cooney et al, 2018). The extent to which these phonological processes are existing in our preparation/imagination stage is not clear and would require specifically designed studies to dissociate them, if possible.…”

“…The current focus of neural decoding has been on either overt speech (Dash et al, 2018d;Livezey et al, 2019) or imagined (covert) speech, which corresponds to imagining speech pronunciation in the absence of articulatory and acoustic output (D'Zmura et al, 2009;Yoshimura et al, 2016;Rezazadeh Sereshkeh et al, 2017;Cooney et al, 2018). Considering the behavioral difficulty in investigating imagined speech, it is understandable that the majority of the speech-BCI research is dominated by overt speech decoding studies.…”

“…In fact, the current decoding studies involving open-set brain to text (Herff et al, 2015;Moses et al, 2019) or brain to speech (Angrick et al, 2019;Anumanchipalli et al, 2019) decoding are on overt speech. Current neural decoding of imagined or intended speech is still limited to closed-set classifications (Guenther et al, 2009;Brumberg et al, 2011;Ikeda et al, 2014;Nguyen et al, 2017;Cooney et al, 2018). For instance, using EEG, researchers have successfully performed imagined speech decoding by classifying various short speech units, e.g., two syllables (D'Zmura et al, 2009), five phonemes (Chi and John, 2011), two vowels (Iqbal et al, 2015;Yoshimura et al, 2016), seven phonemes (Zhao and Rudzicz, 2015), and even words (Porbadnigk et al, 2009;Nguyen et al, 2017;Rezazadeh Sereshkeh et al, 2017;Hashim et al, 2018).…”

“…The neurolinguistics underpinnings supporting phrase-level covert or overt articulation is widely studied topic, but has not yet been explored in a decoding experiment (Memarian et al, 2012). Furthermore, acknowledging the difficulty in verifying the behavioral compliance of imagined speech production (Cooney et al, 2018), in contrast to the data acquisition paradigm of current literature for separately collecting data for overt and imagined speech, we collected the neural signals corresponding to imagined and overt speech consecutively, within the same trial, where the timing of this paradigm constrained the subjects to imagining/preparing the same phrase he/she is expected to articulate for the trial.…”