Decoding speech perception from non-invasive brain recordings

Abstract

Decoding speech from brain activity is a long-awaited goal in both healthcare and neuroscience. Invasive devices have recently led to major milestones in this regard: deep-learning algorithms trained on intracranial recordings can now start to decode elementary linguistic features such as letters, words and audio-spectrograms. However, extending this approach to natural speech and non-invasive brain recordings remains a major challenge. Here we introduce a model trained with contrastive learning to decode self-supervised representations of perceived speech from the non-invasive recordings of a large cohort of healthy individuals. To evaluate this approach, we curate and integrate four public datasets, encompassing 175 volunteers recorded with magneto-encephalography or electro-encephalography while they listened to short stories and isolated sentences. The results show that our model can identify, from 3 seconds of magneto-encephalography signals, the corresponding speech segment with up to 41% accuracy out of more than 1,000 distinct possibilities on average across participants, and with up to 80% in the best participants—a performance that allows the decoding of words and phrases absent from the training set. The comparison of our model with a variety of baselines highlights the importance of a contrastive objective, pretrained representations of speech and a common convolutional architecture simultaneously trained across multiple participants. Finally, the analysis of the decoder’s predictions suggests that they primarily depend on lexical and contextual semantic representations. Overall, this effective decoding of perceived speech from non-invasive recordings delineates a promising path to decode language from brain activity, without putting patients at risk of brain surgery.

Group‐level brain decoding with deep learning

“…Our results expand this work to the task of decoding images from MEG data and provide additional insight into how deep learning and subject embeddings help group-level decoding models. In concurrent work, Défossez et al (2022) have also shown the effectiveness of subject embeddings in group-level speech decoding.…”

“…Our results expand this work to the task of decoding images from MEG data and provide additional insight into how deep learning and subject embeddings help group‐level decoding models. In concurrent work, Défossez et al (2022) have also shown the effectiveness of subject embeddings in group‐level speech decoding. They have also compared it to subject‐specific layers as a way of dealing with between‐subject performance and found this latter approach slightly better.…”

ChineseEEG: A Chinese Linguistic Corpora EEG Dataset for Semantic Alignment and Neural Decoding

“…By analyzing such neural activities, we can uncover the encoding mechanisms of semantics in the brain 3 . A variety of neural signals, including EEG, Functional Magnetic Resonance Imaging (fMRI), Electrocorticography (ECoG) are employed in language-related tasks, from academic research like investigating language processing in the brain to practical applications like language decoding in BCI [4][5][6][7][8][9] . Recently, a lot of studies on neurolinguistics utilized both machine learning methods and modern deep learning methods in NLP to explore linguistic-related problems [10][11][12][13][14][15][16] .…”

“…This mathematical technique transforms the time-domain signal into the frequency domain, revealing the spectrum of frequencies present in the neural recordings. We defined frequency bands of interest-Theta (4-8 Hz), Alpha (8)(9)(10)(11)(12) Hz), Beta (12)(13)(14)(15)(16)(17)(18)(19)(20)(21)(22)(23)(24)(25)(26)(27)(28)(29)(30), and Gamma (30-100 Hz)-to categorize the neural oscillations according to their respective frequency ranges.…”

Context and Attention Shape Electrophysiological Correlates of Speech-to-Language Transformation

“…To gain further insight into the nature of EEG predictions based on Whisper’s explicit transformation of speech-to-language, we ran comparative analyses against two self-supervised speech models, that are trained entirely on sound data, with no access to language. For this, we selected Wav2Vec2 (Baevski et al 2020) and HuBERT (Hsu et al 2021) that in different studies have provided high accuracy fMRI models (Vaidya et al 2022, Millet et al 2022) and a strong basis for decoding model features from MEG and/or EEG data (Défossez et al 2022, Han et al 2023).…”