Direct Speech Reconstruction from Sensorimotor Brain Activity with Optimized Deep Learning Models

Abstract

Development of brain-computer interface (BCI) technology is key for enabling communication in individuals who have lost the faculty of speech due to severe motor paralysis. A BCI control strategy that is gaining attention employs speech decoding from neural data. Recent studies have shown that a combination of direct neural recordings and advanced computational models can provide promising results. Understanding which decoding strategies deliver best and directly applicable results is crucial for advancing the field. In this paper, we optimized and validated a decoding approach based on speech reconstruction directly from high-density electrocorticography recordings from sensorimotor cortex during a speech production task. We show that 1) dedicated machine learning optimization of reconstruction models is key for achieving the best reconstruction performance; 2) individual word decoding in reconstructed speech achieves 92-100\% accuracy (chance level is 8\%); 3) direct reconstruction from sensorimotor brain activity produces intelligible speech. These results underline the need for model optimization in achieving best speech decoding results and highlight the potential that reconstruction-based speech decoding from sensorimotor cortex can offer for development of next-generation BCI technology for communication.

Synthesizing Speech from Intracranial Depth Electrodes using an Encoder-Decoder Framework

“…For the Pearson correlation, each mel-scaled spectral bin is correlated over time individually and then the average is taken. While the Pearson correlation is not a perfect measure of speech quality, it was recently shown to better correlate with intelligibility then other measures [46]. In comparison to existing works, our approach outperforms the baseline from [16] both in terms of mean-squared-error loss (Tab.…”

Cuaderno n.º 2. Mens iura fundamentalia: la neurotecnología ante la Constitución

“…-Resulta esencial entrenar cuidadosamente los modelos de aprendizaje automático para la reconstrucción del habla natural de cara a obtener los mejores resultados. En este sentido, la decodificación basada en la reconstrucción a partir del córtex sensoriomotor podría ser un enfoque prometedor para desarrollar la próxima generación de tecnología BCI para la comunicación 31 .…”

Continuous synthesis of artificial speech sounds from human cortical surface recordings during silent speech production

“…Combining various designs proposed in previous studies [41][42][43], three separate tasks were implemented in the following order: syllable counting (60 sounds), word recognition (60 sounds), and gender recognition (24 sounds). In the first task, volunteers reported the perceived number of syllables, choosing between 1 and 5.…”

“…In terms of model architecture, LDA classifiers combined with spectrogram inversion constituted the most practical workaround to overcome our clinical constraints and minimize model calibration time at the patient's bedside. However, as outlined in a review [54], several recent studies have proposed deep learning approaches that significantly improved the quality of artificial sounds synthesized from intracranial recordings [41,43,50,55,56]. These deep neural networks have not yet been applied both in real time and during silent speech production, but there is no doubt that they would lead to more intelligible artificial sounds.…”