Optimizing deep video representation to match brain activity

Richard, Hugo; Pinho, Ana Luısa; Thirion, Bertrand; Charpiat, Guillaume

doi:10.32470/ccn.2018.1071-0

Cited by 3 publications

(3 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Deep networks trained for image classification give rise to a valuable feature representation. This representation have been shown to closely mimic the hierarchy of the human visual system [1,2,3,4,5,6,7].…”

Section: Introductionmentioning

confidence: 93%

Learning to aggregate feature representations

Gaziv

2019

Preprint

View full text Add to dashboard Cite

The Algonauts challenge requires to construct a multi-subject encoder of images to brain activity. Deep networks such as ResNet-50 and AlexNet trained for image classification are known to produce feature representations along their intermediate stages which closely mimic the visual hierarchy. However the challenges introduced in the Algonauts project, including combining data from multiple subjects, relying on very few similarity data points, solving for various ROIs, and multi-modality, require devising a flexible framework which can efficiently accommodate them. Here we build upon a recent state-of-the-art classification network (SE-ResNeXt-50) and construct an adaptive combination of its intermediate representations. While the pretrained network serves as a backbone of our model, we learn how to aggregate feature representations along five stages of the network. During learning, our method enables to modulate and screen outputs from each stage along the network as governed by the optimized objective. We applied our method to the Algonauts2019 fMRI and MEG challenges. Using the combined fMRI and MEG data, our approach was rated among the leading five for both challenges. Surprisingly we find that for both the lower and higher order areas (EVC and IT) the adaptive aggregation favors features stemming at later stages of the network.

show abstract

Section: Introductionmentioning

confidence: 93%

Learning to aggregate feature representations

Gaziv

2019

Preprint

View full text Add to dashboard Cite

show abstract

“…Such a design matrix encodes the temporal events that affect brain signals during an experiment; these events typically reflect the occurrence of some features of interest in the stimuli. Although some works have used manual annotations (2) or deep neural networks (3,4), see also https://neuroscout.org, to create an estimate of the design matrix of naturalistic stimuli, it is a hard task. Without a design matrix, models such as the general linear model ( 5) cannot be used.…”

Section: Introductionmentioning

confidence: 99%

FastSRM: A fast, memory efficient and identifiable implementation of the shared response model

Richard,

Thirion

2023

Aperture Neuro

View full text Add to dashboard Cite

The shared response model (SRM) provides a simple but effective framework to analyze fMRI data of subjects exposed to naturalistic stimuli. However, when the number of subjects or runs is large, fitting the model requires a large amount of memory and computational power, which limits its use in practice. Furthermore, SRM is not identifiable, which makes the shared response difficult to interpret. In this work, we implement an identifiable version of SRM and show on real data that it improves the stability of the recovered shared response. We then introduce FastSRM, which relies on a dimension reduction step and yields the same solution as the original algorithm. We show experimentally using synthetic and real fMRI data that FastSRM is considerably faster and more memory efficient than current implementations. The experiments performed in this article are fully reproducible: our code available at https://github.com/hugorichard/FastSRM allows you to download the data, run the experiments and plot the figures. For a more pleasant reading of the math equations, we advise the reader to read the version of this article formatted with latex:https://github.com/hugorichard/FastSRM/blob/master/paper/main.pdf.

show abstract

“…The feature representation is chosen such that it closely mimics the neural activity in the visual cortex, with the hope that a simple model (like linear regression) will suffice to capture the remaining mapping. Methods in the second category benefited from utilizing data-driven learned features from leading CNN models trained for natural image classification [5,11,12,13,7,14,15]. The last category refers to recent attempts to train high-complexity deep models which directly decode an fMRI recording into its corresponding image stimulus.…”

Section: Introductionmentioning

confidence: 99%

From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI

Beliy,

Gaziv,

Hoogi

et al. 2019

Preprint

View full text Add to dashboard Cite

Reconstructing observed images from fMRI brain recordings is challenging. Unfortunately, acquiring sufficient "labeled" pairs of {Image, fMRI} (i.e., images with their corresponding fMRI responses) to span the huge space of natural images is prohibitive for many reasons. We present a novel approach which, in addition to the scarce labeled data (training pairs), allows to train fMRI-to-image reconstruction networks also on "unlabeled" data (i.e., images without fMRI recording, and fMRI recording without images). The proposed model utilizes both an Encoder network (image-to-fMRI) and a Decoder network (fMRI-to-image). Concatenating these two networks back-to-back (Encoder-Decoder & Decoder-Encoder) allows augmenting the training with both types of unlabeled data. Importantly, it allows training on the unlabeled test-fMRI data. This self-supervision adapts the reconstruction network to the new input test-data, despite its deviation from the statistics of the scarce training data. 2 Project Website: http://www.wisdom.weizmann.ac.il/~vision/ssfmri2im/ * Equal contribution 2 We will make our code publicly available upon publication.Preprint. Under review.

show abstract

Optimizing deep video representation to match brain activity

Cited by 3 publications

References 11 publications

Learning to aggregate feature representations

Learning to aggregate feature representations

FastSRM: A fast, memory efficient and identifiable implementation of the shared response model

From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI

Contact Info

Product

Resources

About