Identifying Input Features for Development of Real-Time Translation of Neural Signals to Text

Sheth, Janaki; Tankus, Ariel; Tran, Michelle; Comstock, Lindy B; Fried, Itzhak; Speier, William

doi:10.21437/interspeech.2019-3092

Cited by 2 publications

(2 citation statements)

References 14 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Correspondingly, the percentage of channels removed per subject was ∼8%-9%. The localization and relevance of the remaining electrodes are briefly studied both, in the discussion section of this paper using an ablation study as well as, in a previous paper (Sheth et al 2019). In the latter we also address the different parts of speech that these electrodes encode.…”

Section: Experimental Designmentioning

confidence: 99%

“…A frequency band as referred to in this manuscript, is the band-pass filtered oscillatory activity of the group of neurons probed using our depth electrodes. The motivation for such an experiment was that previous studies (Chang et al 2013, Herff et al 2015, Moses et al 2016 have used bands up to high gamma (70-150 Hz) to map unto underlying speech, but a preliminary analysis found tuning in a greater range of frequencies for several of our subjects (figure 2) (Sheth et al 2019).…”

Section: Feature Selectionmentioning

confidence: 99%

See 1 more Smart Citation

Generalizing neural signal-to-text brain-computer interfaces

Sheth

Tankus

Tran

et al. 2021

Biomed. Phys. Eng. Express

Self Cite

View full text Add to dashboard Cite

Objective: Brain-Computer Interfaces (BCI) may help patients with faltering communication abilities due to neurodegenerative diseases produce text or speech by direct neural processing. However, their practical realization has proven difficult due to limitations in speed, accuracy, and generalizability of existing interfaces. The goal of this study is to evaluate the BCI performance of a robust speech decoding system that translates neural signals evoked by speech to a textual output. While previous studies have approached this problem by using neural signals to choose from a limited set of possible words, we employ a more general model that can type any word from a large corpus of English text. Approach: In this study, we create an end-to-end BCI that translates neural signals associated with overt speech into text output. Our decoding system first isolates frequency bands in the input depth-electrode signal encapsulating differential information regarding production of various phonemic classes. These bands form a feature set that then feeds into a Long Short-Term Memory (LSTM) model which discerns at each time point probability distributions across all phonemes uttered by a subject. Finally, a particle filtering algorithm temporally smooths these probabilities by incorporating prior knowledge of the English language to output text corresponding to the decoded word. The generalizability of our decoder is driven by the lack of a vocabulary constraint on this output word. Main result: This method was evaluated using a dataset of 6 neurosurgical patients implanted with intra-cranial depth electrodes to identify seizure foci for potential surgical treatment of epilepsy. We averaged 32% word accuracy and on the phoneme-level obtained 46% precision, 51% recall and 73.32% average phoneme error rate while also achieving significant increases in speed when compared to several other BCI approaches. Significance: Our study employs a more general neural signal-to-text model which could facilitate communication by patients in everyday environments.

show abstract

Section: Experimental Designmentioning

confidence: 99%