Decoding of Chinese phoneme clusters using ECoG

Objective. The goal of this study is to decode the electrical activity of single neurons in the human subthalamic nucleus (STN) to infer the speech features that a person articulated, heard or imagined. We also aim to evaluate the amount of subthalamic neurons required for high accuracy decoding suitable for real-life speech brain-machine interfaces. Approach. We intraoperatively recorded single-neuron activity in the STN of 21 neurosurgical patients with Parkinson's disease undergoing implantation of deep brain stimulator (DBS) while patients produced, perceived or imagined the five monophthongal vowel sounds. Our decoder is based on machine learning algorithms that dynamically learn specific features of the speech-related firing patterns. Main results. In an extensive comparison of algorithms, our sparse decoder ("SpaDe"), based on sparse decomposition of the high dimensional neuronal feature space, outperformed the other algorithms in all three conditions: production, perception and imagery. For speech production, our algorithm, Spade, predicted all vowels correctly (accuracy: 100%; chance level: 20%). For perception accuracy was 96%, and for imagery: 88%. The accuracy of Spade showed a linear behavior in the amount of neurons for the perception data, and even faster for production or imagery. Significance. Our study demonstrates that the information encoded by single neurons in the STN about the production, perception and imagery of speech is suitable for high-accuracy decoding. It is therefore an important step towards brain-machine interfaces for restoration of speech faculties that bears an enormous potential to alleviate the suffering of completely paralyzed ("locked-in") patients and allow them to communicate again with their environment. Moreover, our research indicates how many subthalamic neurons may be necessary to achieve each level of decoding accuracy, which is of supreme importance for a neurosurgeon planning the implantation of a speech brain-machine interface.

Section: Introductionmentioning

confidence: 99%

Machine learning algorithm for decoding multiple subthalamic spike trains for speech brain–machine interfaces

Tankus¹,

Solomon²,

Aharony³

et al. 2021

“…In several studies, electrocorticography (ECoG) was decoded offline to infer parts of speech, for example vowels [26] or complete words [27][28][29][30]. However, these studies made use of a small dictionary for decoding.…”

Section: Introductionmentioning

confidence: 99%

Machine learning decoding of single neurons in the thalamus for speech brain-machine interfaces

Tankus,

Rosenberg,

Ben-Hamo

et al. 2024

Objective. Our goal is to decode firing patterns of single neurons in the left ventralis intermediate nucleus (Vim) of the thalamus, related to speech production, perception, and imagery. For realistic speech brain-machine interfaces (BMIs), we aim to characterize the amount of thalamic neurons necessary for high accuracy decoding.Approach. We intraoperatively recorded single neuron activity in the left Vim of 8 neurosurgical patients undergoing implantation of deep brain stimulator or RF lesioning during production, perception and imagery of the five monophthongal vowel sounds. We utilized the Spade decoder, a machine learning algorithm that dynamically learns specific features of firing patterns and is based on sparse decomposition of the high dimensional feature space.Main results. Spade outperformed all algorithms compared with, for all three aspects of speech: production, perception and imagery, and obtained accuracies of 100%, 96%, and 92%, respectively (chance level: 20%) based on pooling together neurons across all patients. The accuracy was logarithmic in the amount of neurons for all three aspects of speech. Regardless of the amount of units employed, production gained highest accuracies, whereas perception and imagery equated with each other.Significance. Our research renders single neuron activity in the left Vim a promising source of inputs to BMIs for restoration of speech faculties for locked-in patients or patients with anarthria or dysarthria to allow them to communicate again. Our characterization of how many neurons are necessary to achieve a certain decoding accuracy is of utmost importance for planning BMI implantation.

“…Recently, researchers advanced this line of investigation by incorporating a 125 000-word language model, obtaining an error rate of 23.8% using a decoding algorithm comprising the recurrent neural network (RNN) and a Viterbi-based decoding strategy [14]. With ECoG, various decoding targets have been investigated, ranging from vowels/consonants [15][16][17], phonemes [6,[18][19][20][21], words [22][23][24][25][26][27], and sentences [28,29], to articulatory gestures [30][31][32]. For direct speech synthesizing using ECoG, one comparative study combined the recent advances in deep learning with the latest innovations in speech synthesis technologies to reconstruct closed-set intelligible speech from the human auditory cortex [33].…”

Section: Introductionmentioning

confidence: 99%

Speech decoding from stereo-electroencephalography (sEEG) signals using advanced deep learning methods

Wu,

Wellington,

et al. 2024

Objective. Brain–computer interfaces (BCIs) are technologies that bypass damaged or disrupted neural pathways and directly decode brain signals to perform intended actions. BCIs for speech have the potential to restore communication by decoding the intended speech directly. Many studies have demonstrated promising results using invasive micro-electrode arrays and electrocorticography. However, the use of stereo-electroencephalography (sEEG) for speech decoding has not been fully recognized. Approach. In this research, recently released sEEG data were used to decode Dutch words spoken by epileptic participants. We decoded speech waveforms from sEEG data using advanced deep-learning methods. Three methods were implemented: a linear regression method, an recurrent neural network (RNN)-based sequence-to-sequence model (RNN), and a transformer model. Main results. Our RNN and transformer models outperformed the linear regression significantly, while no significant difference was found between the two deep-learning methods. Further investigation on individual electrodes showed that the same decoding result can be obtained using only a few of the electrodes. Significance. This study demonstrated that decoding speech from sEEG signals is possible, and the location of the electrodes is critical to the decoding performance.