Extracting Different Levels of Speech Information from EEG Using an LSTM-Based Model

Monesi, Mohammad Jalilpour; Accou, Bernd; Francart, Tom; hamme, Hugo Van

doi:10.21437/interspeech.2021-336

Cited by 15 publications

(23 citation statements)

References 19 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Throughout the recording session, participants were given short breaks. A subset of this dataset was also used in Accou et al 11,12 , Monesi et al 9,10 and Bollens et al 24 , and is available for the Auditory EEG decoding challenge (https:// exporl. github.…”

Section: Cnnmentioning

confidence: 99%

“…by decoding speech from brain signals [5][6][7] , or by translating both brain signal and speech features to a similar representation 8,9 . Neural tracking in EEG has been found for multiple acoustic representations of speech, such as the spectrogram 7,10 or envelope representations 5,6,11,12 . Additionally, neural tracking has been shown for higher order representations such as semantic dissimilarity, cohort entropy, word surprisal, and phoneme surprisal [13][14][15][16] .…”

mentioning

confidence: 99%

See 1 more Smart Citation

Decoding of the speech envelope from EEG using the VLAAI deep neural network

Accou

Vanthornhout

hamme

et al. 2023

Sci Rep

Self Cite

View full text Add to dashboard Cite

To investigate the processing of speech in the brain, commonly simple linear models are used to establish a relationship between brain signals and speech features. However, these linear models are ill-equipped to model a highly-dynamic, complex non-linear system like the brain, and they often require a substantial amount of subject-specific training data. This work introduces a novel speech decoder architecture: the Very Large Augmented Auditory Inference (VLAAI) network. The VLAAI network outperformed state-of-the-art subject-independent models (median Pearson correlation of 0.19, p < 0.001), yielding an increase over the well-established linear model by 52%. Using ablation techniques, we identified the relative importance of each part of the VLAAI network and found that the non-linear components and output context module influenced model performance the most (10% relative performance increase). Subsequently, the VLAAI network was evaluated on a holdout dataset of 26 subjects and a publicly available unseen dataset to test generalization for unseen subjects and stimuli. No significant difference was found between the default test and the holdout subjects, and between the default test set and the public dataset. The VLAAI network also significantly outperformed all baseline models on the public dataset. We evaluated the effect of training set size by training the VLAAI network on data from 1 up to 80 subjects and evaluated on 26 holdout subjects, revealing a relationship following a hyperbolic tangent function between the number of subjects in the training set and the performance on unseen subjects. Finally, the subject-independent VLAAI network was finetuned for 26 holdout subjects to obtain subject-specific VLAAI models. With 5 minutes of data or more, a significant performance improvement was found, up to 34% (from 0.18 to 0.25 median Pearson correlation) with regards to the subject-independent VLAAI network.

show abstract

Section: Cnnmentioning

confidence: 99%

mentioning

confidence: 99%

Decoding of the speech envelope from EEG using the VLAAI deep neural network

Accou

Vanthornhout

hamme

et al. 2023

Sci Rep

Self Cite

View full text Add to dashboard Cite

show abstract

“…A subset of this dataset was also used in Accou et al 5,6 , Monesi et al 4,20 and Bollens et al 21 This dataset contains approximately 188 hours of EEG recordings (on average 1 hour and 46 minutes per subject) in total.…”

Section: Datasetmentioning

confidence: 99%

“…Recent research has focused on detecting neural tracking of speech features in EEG to understand how speech is procesed by the brain [1][2][3] . Neural tracking has been found for multiple acoustic representations of speech, such as the spectrogram 2,4 or envelope representations 1,3,5,6 . Additionally, neural tracking has been shown for higher order representations such as semantic dissimilarity, cohort entropy, word surprisal, and phoneme surprisal [7][8][9][10] .…”

Section: Introductionmentioning

confidence: 99%

Decoding of the speech envelope from EEG using the VLAAI deep neural network

Accou

Vanthornhout

hamme

et al. 2022

Preprint

Self Cite

View full text Add to dashboard Cite

To investigate the processing of speech in the brain, commonly simple linear models are used to establish a relationship between brain signals and speech features. However, these linear models are ill-equipped to model a highly-dynamic, complex non-linear system like the brain, and they often require a substantial amount of subject-specific training data. This work introduces a novel speech decoder architecture: the Very Large Augmented Auditory Inference (VLAAI) network. The VLAAI network outperformed state-of-the-art subject-independent models (median Pearson correlation of 0.19, p < 0.001), yielding an increase over the well-established linear model by 52%. Using ablation techniques we identified the relative importance of each part of the VLAAI network and found that the non-linear components and output context module influenced model performance the most (10% relative performance increase). Subsequently, the VLAAI network was evaluated on a holdout dataset of 26 subjects and publicly available unseen dataset to test generalization for unseen subjects and stimuli. No significant difference was found between the holdout subjects and the default test set, and only a small difference between the default test set and the public dataset was found. Compared to the baseline models, the VLAAI network still significantly outperformed all baseline models on the public dataset. We evaluated the effect of training set size by training the VLAAI network on data from 1 up to 80 subjects and evaluated on 26 holdout subjects, revealing a logarithmic relationship between the number of subjects in the training set and the performance on unseen subjects. Finally, the subject-independent VLAAI network was fine-tuned for 26 holdout subjects to obtain subject-specific VLAAI models. With 5 minutes of data or more, a significant performance improvement was found, up to 34% (from 0.18 to 0.25 median Pearson correlation) with regards to the subject-independent VLAAI network.

show abstract

“…This paradigm can also be solved in a non-linear fashion with neural networks (e.g. Accou et al, 2021;Monesi et al, 2021;Bollens et al, 2022). Accou et al (2021) showed that the accuracy of a neural network solving a match-mismatch task could be used to estimate the speech reception threshold.…”

Section: Other Methods To Extract Neural Trackingmentioning

confidence: 99%