Mental image reconstruction from human brain activity

Abstract

Visual images perceived by humans can be reconstructed from their brain activity. However, visualization (externalization) of mental images remains challenging. In this study, we demonstrated that our previous image reconstruction method heavily relies on low-level visual information decoded from the brain and cannot efficiently utilize semantic information that would be recruited during mental imagery. To address this limitation, we extended our previous reconstruction method as a Bayesian estimation framework and introduced the assistance of semantic information into it. Our proposed framework successfully reconstructed both seen and imagined images from the brain activity. Interestingly, we found that line components in imagery reconstructions of geometric shapes were emphasized compared to those in seen image reconstructions, which might reflect the sharpening effect caused by the top-down process in the brain. These results suggest that our framework would provide a fundamental technology for directly investigating the subjective contents of the brain.

Visualizing the mind's eye: a future perspective on applications of image reconstruction from brain signals to psychiatry

“…observed images) from brain activity, some studies have also successfully reconstructed subjective visual experience in the human mind (e.g. attention, memory, mental imagery, illusory perception) (Cheng et al ., 2023 ; Horikawa & Kamitani, 2022 ; Koide-Majima et al ., 2023 ; Lee & Kuhl, 2016 ; Shen, Horikawa, et al ., 2019 ; Shimizu & Srinivasan, 2022 ), which remains a challenge to be actively explored in the field of cognitive computational neuroscience, but also provides vast prospect in computational psychiatry.…”