Extracting multidimensional stimulus-response correlations using hybrid encoding-decoding of neural activity

Dmochowski, Jacek; Ki, Jason J.; DeGuzman, Paul; Sajda, Paul; Parra, Lucas C.

doi:10.1016/j.neuroimage.2017.05.037

Cited by 64 publications

(102 citation statements)

References 73 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…One is the fronto-central region, and the other is the parietal-occipital region. This is in line with other related works that estimate spatial filters based on speech processing (Dmochowski et al, 2018) and those that shows the topography of speech processing in the brain Hjortkjaer et al, 2018;O'Sullivan et al, 2015). For the second component we did not find any significant clusters.…”

Section: ) Data Analysissupporting

confidence: 93%

“…This, however, requires repeated trials, which renders it impractical for many EEG applications (although for MEG data, a few trials are typically enough for DSS to obtain a useful dimensionality reduction and denoising (Akram et al, 2016(Akram et al, , 2017Ding et al, 2014)). Canonical correlation analysis (CCA) also reduces dimensionality by finding separate linear transformations for the stimulus as well as neural responses, such that in the respective projected subspaces, the neural response and the stimulus are maximally correlated (de Cheveigné et al, 2018a,b;Dmochowski et al, 2018;Hotelling, 1936).…”

mentioning

confidence: 99%

See 1 more Smart Citation

Stimulus-aware spatial filtering for single-trial neural response and temporal response function estimation in high-density EEG with applications in auditory research

Das

Vanthornhout

Francart

et al. 2019

Preprint

View full text Add to dashboard Cite

A common problem in neural recordings is the low signal-to-noise ratio (SNR), particularly when using noninvasive techniques like magneto-or electroencephalography (M/EEG). To address this problem, experimental designs often include repeated trials, which are then averaged to improve the SNR or to inform the design of a spatial filter that projects the data onto high-SNR directions. However, collecting enough repeated trials is often impractical and even impossible in some paradigms. Therefore, we present a data-driven spatial filter design that takes advantage of the knowledge of the presented stimulus, to achieve a joint noise reduction and dimensionality reduction without the need for repeated trials. The method uses the stimulus-driven neural response, which is then used to find a set of spatial filters that maximize the SNR based on a generalized eigenvalue decomposition. As the method is fully data-driven, the dimensionality reduction enables researchers to perform their analyses without having to rely on their knowledge of brain regions of interest, which increases accuracy and reduces the human factor in the results. In the context of neural tracking of a speech stimulus, our method resulted in better short-term temporal response function (TRF) estimates, higher correlations between predicted and actual neural responses, and higher attention decoding accuracies compared to existing TRF-based decoding methods. We also provide an extensive discussion on the central role played by the generalized eigenvalue decomposition in various denoising methods in the literature, and address the conceptual similarities and differences with our proposed method.

show abstract

Section: ) Data Analysissupporting

confidence: 93%

mentioning

confidence: 99%

Stimulus-aware spatial filtering for single-trial neural response and temporal response function estimation in high-density EEG with applications in auditory research

Das

Vanthornhout

Francart

et al. 2019

Preprint

View full text Add to dashboard Cite

show abstract

“…The association between the fMRI data and the deep features fc7 (see multivariate linking box in Figure 1) can be learnt using multivariate linking methods. Canonical Correlation Analysis (CCA) [23] is often used in this respect [40,41,42,43,44], as it allows projecting one dataset onto another by means of linear mapping, which can be further used for categorical discrimination and brain model interpretations.…”

Section: Linking Methodsmentioning

confidence: 99%

Transfer learning of deep neural network representations for fMRI decoding

Svanera

Savardi

Benini

et al. 2019

Preprint

View full text Add to dashboard Cite

BackgroundDeep neural networks have revolutionised machine learning, with unparalleled performance in object classification. However, in brain imaging (e.g. fMRI), the direct application of Convolutional Neural Networks (CNN) to decoding subject states or perception from imaging data seems impractical given the scarcity of available data. New methodIn this work we propose a robust method to transfer information from deep learning (DL) features to brain fMRI data with the goal of decoding. By adopting Reduced Rank Regression with Ridge Regularisation we establish a multivariate link between imaging data and the fully connected layer (fc7) of a CNN. We exploit the reconstructed fc7 features by performing an object image classification task on two datasets: one of the largest fMRI databases, taken from different scanners from more than two hundred subjects watching different movie clips, and another with fMRI data taken while watching static images, ResultsThe fc7 features could be significantly reconstructed from the imaging data, and led to significant decoding performance. Comparison with existing methodsThe decoding based on reconstructed fc7 outperformed the decoding based on imaging data alone. ConclusionIn this work we show how to improve fMRI-based decoding benefiting from the mapping between functional data and CNN features. The potential advantage of the proposed method is twofold: the extraction of stimuli representations by means of an automatic procedure (unsupervised) and the embedding of highdimensional neuroimaging data onto a space designed for visual object discrimination, leading to a more manageable space from dimensionality point of view.A long-standing goal of cognitive neuroscience is to unravel the brain mechanisms associated with sensory perception. Cognitive neuroscientists often conduct empirical research using non-invasive imaging techniques, among which functional Magnetic Resonance Imaging (fMRI) or Electroencephalography (EEG), to validate computational theories and models by relating sensory experiences, like watching images and videos, to the observed brain activity. Establishing such relationship is not trivial, due to our partial understanding of the neural mechanisms involved, the limited view offered by current imaging techniques, and the high dimensions in both imaging and sensorial spaces.A large amount of statistical approaches have been proposed in the literature to accomplish this task; in particular, in the last two decades great attention has been given to generative (also referred to as encoding) and discriminative (decoding) models, that have different aims, strengths and limitations (see [1]). Encoding models aim at characterising single units response harnessing the richness of the stimulus representation in a suitable space, and can thus be used to model the brain response to new stimuli, provided that a suitable decomposition is available. On the other hand, decoding models solve a "simpler" problem of discriminating between specific stimulus types and are better sui...

show abstract

“…We demonstrated that the brain responses to naturalistic speech could be better predicted by including a phonemebased stimulus representation, and by extracting speech-related information from the recorded EEG signals using a GEVD approach combined with standard multivariate TRF (Crosse et al, 2016). This method, which is closely related to the CCA-based method proposed by de Cheveigné & Parra (2014) and Dmochowski, Ki, DeGuzman, Sajda, & Parra (2018), allows substantially increasing the EEG prediction, with a correlation of around 0.1 reached using the delta EEG band and a high-level speech representation. Moreover, we here showed that the inclusion of this linear spatial mapping allows removing the impact of manual channel selection on the performance by automatically weighting channels in a data-driven way.…”

Section: Consistent With DImentioning

confidence: 95%

Data-driven spatial filtering for improved measurement of cortical tracking of multiple representations of speech

Lesenfants

Vanthornhout

Verschueren

et al. 2019

Preprint

View full text Add to dashboard Cite

Objective -Measurement of the cortical tracking of continuous speech from electroencephalography (EEG) recordings using a forward model is an important tool in auditory neuroscience. Usually the stimulus is represented by its temporal envelope. Recently, the phonetic representation of speech was successfully introduced in English. We aim to show that the EEG prediction from phoneme-related speech features is possible in Dutch. The method requires a manual channel selection based on visual inspection or prior knowledge to obtain a summary measure of cortical tracking. We evaluate a method to (1) remove nonstimulus-related activity from the EEG signals to be predicted, and (2) automatically select the channels of interest. Approach -Eighteen participants listened to a Flemish story, while their EEG was recorded. Subject-specific and grand-average temporal response functions were determined between the EEG activity in different frequency bands and several stimulus features: the envelope, spectrogram, phonemes, phonetic features or a combination. The temporal response functions were used to predict EEG from the stimulus, and the predicted was compared with the recorded EEG, yielding a measure of cortical tracking of stimulus features. A spatial filter was calculated based on the generalized eigenvalue decomposition (GEVD), and the effect on EEG prediction accuracy was determined. Main results -A model including both low-and high-level speech representations was able to better predict the brain responses to the speech than a model only including low-level features. The inclusion of a GEVD-based spatial filter in the model increased the prediction accuracy of cortical responses to each speech feature at both single-subject (270% improvement) and group-level (310%). Significance -We showed that the inclusion of acoustical and phonetic speech information and the addition of a data-driven spatial filter allow improved modelling of the relationship between the speech and its brain responses and offer an automatic channel selection.

show abstract

Extracting multidimensional stimulus-response correlations using hybrid encoding-decoding of neural activity

Cited by 64 publications

References 73 publications

Stimulus-aware spatial filtering for single-trial neural response and temporal response function estimation in high-density EEG with applications in auditory research

Stimulus-aware spatial filtering for single-trial neural response and temporal response function estimation in high-density EEG with applications in auditory research

Transfer learning of deep neural network representations for fMRI decoding

Data-driven spatial filtering for improved measurement of cortical tracking of multiple representations of speech

Contact Info

Product

Resources

About