Generative adversarial networks for reconstructing natural images from brain activity

Abstract

We explore a method for reconstructing visual stimuli from brain activity. Using large databases of natural images we trained a deep convolutional generative adversarial network capable of generating gray scale photos, similar to stimuli presented during two functional magnetic resonance imaging experiments. Using a linear model we learned to predict the generative model's latent space from measured brain activity. The objective was to create an image similar to the presented stimulus image through the previously trained generator. Using this approach we were able to reconstruct structural and some semantic features of a proportion of the natural images sets. A behavioural test showed that subjects were capable of identifying a reconstruction of the original stimulus in 67.2% and 66.4% of the cases in a pairwise comparison for the two natural image datasets respectively. Our approach does not require end-to-end training of a large generative model on limited neuroimaging data. Rapid advances in generative modeling promise further improvements in reconstruction performance.

End-to-End Deep Image Reconstruction From Human Brain Activity

“…Earlier studies on decoding stimuli in pixel space either searched for a match in the exemplar set (Naselaris et al, 2009; Nishimoto et al, 2011) or tried to reconstruct the stimulus (Miyawaki et al, 2008; Wen et al, 2016; Güçlütürk et al, 2017; Han et al, 2017; Seeliger et al, 2018; Shen et al, 2019). In the exemplar matching methods, visualization is limited to the samples in the exemplar set and hence these methods cannot be generalized to stimuli that are not included in the exemplar set.…”

“…DNN-based reconstruction methods have typically avoided directly training a DNN model for reconstruction (Güçlütürk et al, 2017; Han et al, 2017; Seeliger et al, 2018; Shen et al, 2019). Instead, they have used decoded features as a proxy for hierarchical visual representations encoded in the fMRI activity that was used as the input to a reconstruction module.…”

“…We evaluated the quality of reconstruction using both objective and subjective assessment methods. For both methods, we performed a pairwise similarity comparison, following previous studies (Cowen et al, 2014; Lee and Kuhl, 2016; Seeliger et al, 2018; Shen et al, 2019), in which one reconstructed image was compared with two candidate images: the original stimulus image from which the reconstruction was derived and a “lure” image, which was a different test image. The lure image was randomly selected from the test dataset of the same type as the original stimulus image.…”

“…Advances in the deep learning have opened new directions to decode and visualize the information present in the human brain. In the past few years, deep neural networks (DNNs) have been successfully used to reconstruct visual content from brain activity measured by functional magnetic resonance imaging (fMRI) (Güçlütürk et al, 2017; Han et al, 2017; Seeliger et al, 2018; Shen et al, 2019).…”

Evaluating the Usability of Passthought Authentication

“…An impressive but foreboding body of research has demonstrated the capability of statistical modelling and machine learning techniques to reconstruct images viewed by a participant undergoing an fMRI scan. [162][163][164][165][166][167] For example, a 2019 study by Shen et al 162 demonstrated a generative adversarial network (GAN) model which produced the images in Figure 2.5. The practical relevance of these findings from a security perspective is lacking, given that fMRIs are currently not very suitable for BCI applications due to their size and cost, and the reproduced images The aim was to build a system to believably emulate the user experience of passthought authentication, rather than to build a fully-functional and properly secure authenticator.…”