BigGAN-based Bayesian Reconstruction of Natural Images from Human Brain Activity

Qiao, Kai; Chen, Jian; Wang, Linyuan; Zhang, Chi; Li, Tong; Yan, Bin

doi:10.1016/j.neuroscience.2020.07.040

Cited by 23 publications

(16 citation statements)

References 54 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…A group of studies by Seeliger et al (2018), Mozafari et al (2020), and Qiao et al (2020) utilized GAN architecture with the assumption that there is a linear relationship between brain activity and the latent features of GAN. Similar to ShenDNN+DGN, these methods adopted the generator of a pretrained GAN as a natural image prior, which ensures that the reconstructed images follow similar distributions as natural images.…”

Section: Generative Adversarial Network (Gan)mentioning

confidence: 99%

“…The GAN-based Bayesian visual reconstruction model (GAN-BVRM) proposed by Qiao et al (2020) aims to improve the quality of reconstructions from a limited dataset combination and, as the name suggests, uses the combination of GAN and Bayesian learning. From Bayesian perspective, a conditional distribution p(v|x) corresponds to an encoder which predicts fMRI activity v from the stimuli image x.…”

Section: Generative Adversarial Network (Gan)mentioning

confidence: 99%

“…However, the definition and implementation of the perceptual similarity metric in terms of the distance metric or feature extraction network vary across studies. For example, Qiao et al (2020) utilized five convolutional layers of the AlexNet (Krizhevsky et al, 2012) to extract hierarchical features. The other study by Mozafari et al (2020), proposed a high-level similarity measure, which measures perceptual similarity based on the output of only the last layer of Inception-V3 (Szegedy et al, 2016).…”

Section: Psmmentioning

confidence: 99%

See 2 more Smart Citations

Natural Image Reconstruction From fMRI Using Deep Learning: A Survey

et al. 2021

View full text Add to dashboard Cite

With the advent of brain imaging techniques and machine learning tools, much effort has been devoted to building computational models to capture the encoding of visual information in the human brain. One of the most challenging brain decoding tasks is the accurate reconstruction of the perceived natural images from brain activities measured by functional magnetic resonance imaging (fMRI). In this work, we survey the most recent deep learning methods for natural image reconstruction from fMRI. We examine these methods in terms of architectural design, benchmark datasets, and evaluation metrics and present a fair performance evaluation across standardized evaluation metrics. Finally, we discuss the strengths and limitations of existing studies and present potential future directions.

show abstract

Section: Generative Adversarial Network (Gan)mentioning

confidence: 99%

Section: Generative Adversarial Network (Gan)mentioning

confidence: 99%

Section: Psmmentioning

confidence: 99%

See 1 more Smart Citation

Natural Image Reconstruction From fMRI Using Deep Learning: A Survey

et al. 2021

View full text Add to dashboard Cite

show abstract

“…Furthermore, the related problem of text to image translation has also been tackled with GANs [31][32][33], although the most astonishing results on that task are from the Dall-E model [10]. Moreover, GANs have been used to translate functional MRI (fMRI) data back to the presented visual stimulus that evoked it [34][35][36][37][38][39][40], which is useful for uncovering internal neural representations.…”

Section: Trends In Cognitive Sciencesmentioning

confidence: 99%

Generative adversarial networks unlock new methods for cognitive science

Goetschalckx

Andonian²,

Wagemans

2021

Trends in Cognitive Sciences

View full text Add to dashboard Cite

“…Reconstructing natural images from fMRI was approached by a number of methods, which can broadly be classified into three families: (i) Linear regression between fMRI data and handcrafted image-features (e.g., Gabor wavelets) [26][27][28], (ii) Linear regression between fMRI data and deep (CNN-based) image-features (e.g., using pretrained AlexNet) [12,[29][30][31], or latent spaces of pretrained generative models [32][33][34][35], and (iii) End-to-end Deep Learning [13,[36][37][38]. To our best knowledge, methods [12] and [36] are the current state-of-the-art in this field.…”

Section: Introductionmentioning

confidence: 99%

Self-Supervised Natural Image Reconstruction and Large-Scale Semantic Classification from Brain Activity

Gaziv

Beliy

Granot

et al. 2020

Preprint

View full text Add to dashboard Cite

Reconstructing natural images and decoding their semantic category from fMRI brain recordings is challenging. Acquiring sufficient pairs (image, fMRI) that span the huge space of natural images is prohibitive. We present a novel self-supervised approach for fMRI-to-image reconstruction and classification that goes well beyond the scarce paired data. By imposing cycle consistency, we train our image reconstruction deep neural network on many “unpaired” data: a plethora of natural images without fMRI recordings (from many novel categories), and fMRI recordings without images. Combining high-level perceptual objectives with self-supervision on unpaired data results in a leap improvement over top existing methods, achieving: (i) Unprecedented image-reconstruction from fMRI of never-before-seen images (evaluated by image metrics and human testing); (ii) Large-scale semantic classification (1000 diverse classes) of categories that are never-before-seen during network training. Such large-scale (1000-way) semantic classification capabilities from fMRI recordings have never been demonstrated before. Finally, we provide evidence for the biological plausibility of our learned model. 1

show abstract

BigGAN-based Bayesian Reconstruction of Natural Images from Human Brain Activity

Cited by 23 publications

References 54 publications

Natural Image Reconstruction From fMRI Using Deep Learning: A Survey

Natural Image Reconstruction From fMRI Using Deep Learning: A Survey

Generative adversarial networks unlock new methods for cognitive science

Self-Supervised Natural Image Reconstruction and Large-Scale Semantic Classification from Brain Activity

Contact Info

Product

Resources

About