High-resolution image reconstruction with latent diffusion models from human brain activity

Abstract

Reconstructing visual experiences from human brain activity offers a unique way to understand how the brain represents the world, and to interpret the connection between computer vision models and our visual system. While deep generative models have recently been employed for this task, reconstructing realistic images with high semantic fidelity is still a challenging problem. Here, we propose a new method based on a diffusion model (DM) to reconstruct images from human brain activity obtained via functional magnetic resonance imaging (fMRI). More specifically, we rely on a latent diffusion model (LDM) termed Stable Diffusion. This model reduces the computational cost of DMs, while preserving their high generative performance. We also characterize the inner mechanisms of the LDM by studying how its different components (such as the latent vector of image Z, conditioning inputs C, and different elements of the denoising U-Net) relate to distinct brain functions. We show that our proposed method can reconstruct high-resolution images with high fidelity in straightforward fashion, without the need for any additional training and fine-tuning of complex deep-learning models. We also provide a quantitative interpretation of different LDM components from a neuroscientific perspective. Overall, our study proposes a promising method for reconstructing images from human brain activity, and provides a new framework for understanding DMs.

Generating realistic neurophysiological time series with denoising diffusion probabilistic models

“…For example, variational autoencoders (VAEs) have been applied to infer low-dimensional representations of single-trial neural population dynamics (Pandarinath et al, 2018), while generative adversarial networks (GANs) have been used for the task of spike-train generation (Molano-Mazon et al, 2018, Ramesh et al, 2019), as well as to decode images from single neuron and fMRI data (Ponce et al, 2019, Lin et al, 2022). The recently proposed denoising diffusion probabilistic models (DDPMs) have also been applied to improve neural decoding performance, in particular leveraging latent diffusion models (Rombach et al, 2022) to predict viewed images from fMRI data (Takagi and Nishimoto, 2022, Chen et al, 2023).…”

Mental image reconstruction from human brain activity

“…Recently, by incorporating the assistance of deep neural networks (DNNs) 12,13 and generative models [14][15][16][17][18][19][20][21] , several studies have achieved higher-fidelity natural image reconstruction [22][23][24][25][26] , which has become a tool for investigating the visual processing in the brain (e.g., visual representation, attention 27 , and illusion 28 ).…”

“…Other studies have decoded seen natural images 9, 10 or videos 11 using visual features inspired by neurophysiological discoveries. Recently, by incorporating the assistance of deep neural networks (DNNs) 12, 13 and generative models 14–21 , several studies have achieved higher-fidelity natural image reconstruction 22–26 , which has become a tool for investigating the visual processing in the brain (e.g., visual representation, attention 27 , and illusion 28 ).…”

Human visual cortex and deep convolutional neural network care deeply about object background

“…Deep convolutional neural networks (DCNNs) have entered the computational modeling scene with high predictive performance of both object category and brain dynamics during object categorization tasks (1)(2)(3)(4). These predictions on brain dynamics are not limited to lowlevel image statistics but also include high-level features such as animacy, object category and semantics (5)(6)(7)(8)(9). In fact, DCNNs' predictive performance on visual processes surpassed hand-engineered, biologically-inspired models (e.g.…”