Increasingly complex representations of natural movies across the dorsal stream are shared between subjects

Güçlü, Umut; Gerven, Marcel A. J. van

doi:10.1016/j.neuroimage.2015.12.036

Cited by 126 publications

(134 citation statements)

References 38 publications

Supporting

Mentioning

121

Contrasting

Order By: Relevance

“…A 3‐D CNN shares the same computational principle as an otherwise 2‐D CNN, except that the input to the former is a time series of video frames with a specific duration, whereas the input to the latter is a single video frame or image. Previously, the 3‐D CNN was shown to explain cortical fMRI responses to natural movie stimuli (Güçlü & van Gerven, ). However, it is unlikely that the brain works in a similar way as a 3‐D CNN.…”

Section: Discussionmentioning

confidence: 99%

Deep recurrent neural network reveals a hierarchy of process memory during dynamic natural vision

Shi

Wen

Zhang

et al. 2018

Human Brain Mapping

View full text Add to dashboard Cite

The human visual cortex extracts both spatial and temporal visual features to support perception and guide behavior. Deep convolutional neural networks (CNNs) provide a computational framework to model cortical representation and organization for spatial visual processing, but unable to explain how the brain processes temporal information. To overcome this limitation, we extended a CNN by adding recurrent connections to different layers of the CNN to allow spatial representations to be remembered and accumulated over time. The extended model, or the recurrent neural network (RNN), embodied a hierarchical and distributed model of process memory as an integral part of visual processing. Unlike the CNN, the RNN learned spatiotemporal features from videos to enable action recognition. The RNN better predicted cortical responses to natural movie stimuli than the CNN, at all visual areas, especially those along the dorsal stream. As a fully observable model of visual processing, the RNN also revealed a cortical hierarchy of temporal receptive window, dynamics of process memory, and spatiotemporal representations. These results support the hypothesis of process memory, and demonstrate the potential of using the RNN for in-depth computational understanding of dynamic natural vision.

show abstract

Section: Discussionmentioning

confidence: 99%

Deep recurrent neural network reveals a hierarchy of process memory during dynamic natural vision

Shi

Wen

Zhang

et al. 2018

Human Brain Mapping

View full text Add to dashboard Cite

show abstract

“…However, encoding models are usually used to analyze data from subjects one[ 3 2 _ T D $ D I F F ] by[ 3 3 _ T D $ D I F F ] one and [ 3 4 _ T D $ D I F F ] do not explicitly assess common features across subjects. The good news is that encoding modeling and hyperalignment frameworks are not exclusive but can be used in a complementary manner (e.g., [9]). Combinations of these frameworks might provide explicit links between a common representational space procedure learns subject-wise optimal transformations of brain activity into a common representational space.…”

Section: [ 2 6 _ T D $ D I F F ]mentioning

confidence: 99%

Lining Up Brains via a Common Representational Space

Nishimoto

Nishida

2016

Trends in Cognitive Sciences

View full text Add to dashboard Cite

“…Decoding studies based on fMRI have matured significantly during the last 15 years (see e.g. [Haynes andRees, 2006, Gerlach, 2007] for review), and human brain activity has been successfully decoded from natural images and movies [Kay et al, 2008, Prenger et al, 2009, Nishimoto et al, 2011, Huth et al, 2012, Huth et al, 2016, Güçlü and van Gerven, 2017.…”

Section: Introductionmentioning

confidence: 99%

Single-Trial Decoding of Scalp EEG Under Natural Conditions

Tuckute

Hansen

Pedersen

et al. 2018

Preprint

View full text Add to dashboard Cite

There is significant current interest in decoding mental states from electro-encephalography (EEG) recordings. EEG signals are subject-specific, sensitive to disturbances, and have a low signal-to-noise ratio, which has been mitigated by the use of laboratory-grade EEG acquisition equipment under highly controlled conditions. In the present study, we investigate single-trial decoding of natural, complex stimuli based on scalp EEG acquired with a portable, 32 dry-electrode sensor system in a typical office setting. We probe generalizability by a leave-one-subject-out cross-validation approach. We demonstrate that Support Vector Machine (SVM) classifiers trained on a relatively small set of de-noised (averaged) pseudo-trials perform on par with classifiers trained on a large set of noisy single-trial samples. For visualization of EEG signatures exploited by SVM classifiers, we propose a novel method for computing sensitivity maps of EEG-based SVM classifiers. Moreover, we apply the NPAIRS resampling framework for estimation of map uncertainty and show that effect sizes of sensitivity maps for classifiers trained on small samples of de-noised data and large samples of noisy data are similar. Finally, we demonstrate that the average pseudo-trial classifier can successfully predict the class of single trials from withheld subjects, which allows for fast classifier training, parameter optimization and unbiased performance evaluation in machine learning approaches for brain decoding.

show abstract

Increasingly complex representations of natural movies across the dorsal stream are shared between subjects

Cited by 126 publications

References 38 publications

Deep recurrent neural network reveals a hierarchy of process memory during dynamic natural vision

Deep recurrent neural network reveals a hierarchy of process memory during dynamic natural vision

Lining Up Brains via a Common Representational Space

Single-Trial Decoding of Scalp EEG Under Natural Conditions

Contact Info

Product

Resources

About