Decoding of the speech envelope from EEG using the VLAAI deep neural network

Abstract

To investigate the processing of speech in the brain, commonly simple linear models are used to establish a relationship between brain signals and speech features. However, these linear models are ill-equipped to model a highly-dynamic, complex non-linear system like the brain, and they often require a substantial amount of subject-specific training data. This work introduces a novel speech decoder architecture: the Very Large Augmented Auditory Inference (VLAAI) network. The VLAAI network outperformed state-of-the-art subject-independent models (median Pearson correlation of 0.19, p < 0.001), yielding an increase over the well-established linear model by 52%. Using ablation techniques we identified the relative importance of each part of the VLAAI network and found that the non-linear components and output context module influenced model performance the most (10% relative performance increase). Subsequently, the VLAAI network was evaluated on a holdout dataset of 26 subjects and publicly available unseen dataset to test generalization for unseen subjects and stimuli. No significant difference was found between the holdout subjects and the default test set, and only a small difference between the default test set and the public dataset was found. Compared to the baseline models, the VLAAI network still significantly outperformed all baseline models on the public dataset. We evaluated the effect of training set size by training the VLAAI network on data from 1 up to 80 subjects and evaluated on 26 holdout subjects, revealing a logarithmic relationship between the number of subjects in the training set and the performance on unseen subjects. Finally, the subject-independent VLAAI network was fine-tuned for 26 holdout subjects to obtain subject-specific VLAAI models. With 5 minutes of data or more, a significant performance improvement was found, up to 34% (from 0.18 to 0.25 median Pearson correlation) with regards to the subject-independent VLAAI network.

Auditory EEG Decoding Challenge for ICASSP 2023

“…Therefore, future work should use richer, more complex speech representations. In addition, finetuning the models for the within-subject test set provides extra accuracy, which is consistent with prior studies [7], [14], [46].…”

“…We include a simple linear model as well as the Very Large Augmented Auditory Inference (VLAAI) network [7] as a baseline for task 2. The linear model is implemented using a one-dimensional convolutional layer in TensorFlow with kernel size of 32 (corresponds to a 500 ms integration window).…”

Attention and Sequence Modeling for Match-Mismatch Classification of Speech Stimulus and EEG Response

“…The data set [57], [58] used in the Auditory EEG Decoding Challenge comprises EEG data that was obtained from 85 young adults who are Dutch native speakers and have normal hearing capabilities. Throughout the experiment, each participant engaged in approximately 8 to 10 trials, with each trial lasting around 15 minutes.…”

SparrKULee: A Speech-evoked Auditory Response Repository of the KU Leuven, containing EEG of 85 participants

“…In order to study the neural processing of speech, recent studies have presented natural running speech to participants while the electroencephalogram (EEG) was recorded. Currently, regression is used to either decode features from the speech stimulus from the EEG (also known as a backward model) [1][2][3][4][5] , to predict the EEG from the speech stimulus 1,6 (forward model), or to transform both EEG and speech stimulus to a shared space 7,8 (hybrid model). Deep neural networks have recently been proposed for auditory decoding and have obtained promising results 4,5,[9][10][11][12] .…”