Learning Subject-Invariant Representations from Speech-Evoked EEG Using Variational Autoencoders

et al. 2021

Preprint

Self Cite

The neural tracking framework enables the analysis of neural responses (EEG) to continuous natural speech, e.g., a story or a podcast. This allows for objective investigation of a range of auditory and linguistic processes in the brain during natural speech perception. This approach is more ecologically valid than traditional auditory evoked responses and has great potential for both research and clinical applications. In this article, we review the neural tracking framework and highlight three prominent examples of neural tracking analyses. This includes the neural tracking of the fundamental frequency of the voice (f0), the speech envelope and linguistic features. Each of these analyses provides a unique point of view into the hierarchical stages of speech processing in the human brain. f0-tracking assesses the encoding of fine temporal information in the early stages of the auditory pathway, i.e. from the auditory periphery up to early processing in the primary auditory cortex. This fundamental processing in (mostly) subcortical stages forms the foundation of speech perception in the cortex. Envelope tracking reflects bottom-up and top-down speech-related processes in the auditory cortex, and is likely necessary but not sufficient for speech intelligibility. To study neural processes more directly related to speech intelligibility, neural tracking of linguistic features can be used. This analysis focuses on the encoding of linguistic features (e.g. word or phoneme surprisal) in the brain. Together these analyses form a multi-faceted and time-effective objective assessment of the auditory and linguistic processing of an individual.

Section: Methods To Measure Neural Trackingmentioning

confidence: 99%

Neural tracking as a diagnostic tool to assess the auditory pathway

Canneyt

Gillis

et al. 2021

Preprint

Self Cite

“…A subset of this dataset was also used in Accou et al 5,6 , Monesi et al 4,20 and Bollens et al 21 This dataset contains approximately 188 hours of EEG recordings (on average 1 hour and 46 minutes per subject) in total.…”

Section: Datasetmentioning

confidence: 99%

“…Deep, non-linear artificial neural networks have been proposed as an alternative over linear models to model the complex non-linear brain 5,[18][19][20][21] . Recently, deep learning methods have been successfully applied to the match/mismatch paradigm 5,20 .…”

Section: Introductionmentioning

confidence: 99%

“…Deep, non-linear artificial neural networks have been proposed as an alternative over linear models to model the complex non-linear brain ( de Taillez et al, 2017 ; Ciccarelli et al, 2019 ; Monesi et al, 2020; Accou et al, 2021a ; Bollens et al, 2022 ). Recently, deep learning methods have been successfully applied to the match/mismatch paradigm ( Monesi et al, 2020 ; Accou et al, 2021a ).…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Decoding of the speech envelope from EEG using the VLAAI deep neural network

Accou

hamme

et al. 2022

Preprint

Self Cite

To investigate the processing of speech in the brain, commonly simple linear models are used to establish a relationship between brain signals and speech features. However, these linear models are ill-equipped to model a highly-dynamic, complex non-linear system like the brain, and they often require a substantial amount of subject-specific training data. This work introduces a novel speech decoder architecture: the Very Large Augmented Auditory Inference (VLAAI) network. The VLAAI network outperformed state-of-the-art subject-independent models (median Pearson correlation of 0.19, p < 0.001), yielding an increase over the well-established linear model by 52%. Using ablation techniques we identified the relative importance of each part of the VLAAI network and found that the non-linear components and output context module influenced model performance the most (10% relative performance increase). Subsequently, the VLAAI network was evaluated on a holdout dataset of 26 subjects and publicly available unseen dataset to test generalization for unseen subjects and stimuli. No significant difference was found between the holdout subjects and the default test set, and only a small difference between the default test set and the public dataset was found. Compared to the baseline models, the VLAAI network still significantly outperformed all baseline models on the public dataset. We evaluated the effect of training set size by training the VLAAI network on data from 1 up to 80 subjects and evaluated on 26 holdout subjects, revealing a logarithmic relationship between the number of subjects in the training set and the performance on unseen subjects. Finally, the subject-independent VLAAI network was fine-tuned for 26 holdout subjects to obtain subject-specific VLAAI models. With 5 minutes of data or more, a significant performance improvement was found, up to 34% (from 0.18 to 0.25 median Pearson correlation) with regards to the subject-independent VLAAI network.

“…Deep, non-linear artificial neural networks have been proposed as an alternative over linear models to model the complex non-linear brain 9,11,[22][23][24] . Recently, deep learning methods have been successfully applied to the VLAAI.…”

mentioning

confidence: 99%

Decoding of the speech envelope from EEG using the VLAAI deep neural network

Accou

hamme

et al. 2023

Sci Rep

Self Cite

To investigate the processing of speech in the brain, commonly simple linear models are used to establish a relationship between brain signals and speech features. However, these linear models are ill-equipped to model a highly-dynamic, complex non-linear system like the brain, and they often require a substantial amount of subject-specific training data. This work introduces a novel speech decoder architecture: the Very Large Augmented Auditory Inference (VLAAI) network. The VLAAI network outperformed state-of-the-art subject-independent models (median Pearson correlation of 0.19, p < 0.001), yielding an increase over the well-established linear model by 52%. Using ablation techniques, we identified the relative importance of each part of the VLAAI network and found that the non-linear components and output context module influenced model performance the most (10% relative performance increase). Subsequently, the VLAAI network was evaluated on a holdout dataset of 26 subjects and a publicly available unseen dataset to test generalization for unseen subjects and stimuli. No significant difference was found between the default test and the holdout subjects, and between the default test set and the public dataset. The VLAAI network also significantly outperformed all baseline models on the public dataset. We evaluated the effect of training set size by training the VLAAI network on data from 1 up to 80 subjects and evaluated on 26 holdout subjects, revealing a relationship following a hyperbolic tangent function between the number of subjects in the training set and the performance on unseen subjects. Finally, the subject-independent VLAAI network was finetuned for 26 holdout subjects to obtain subject-specific VLAAI models. With 5 minutes of data or more, a significant performance improvement was found, up to 34% (from 0.18 to 0.25 median Pearson correlation) with regards to the subject-independent VLAAI network.