‘When’ and ‘what’ did you see? A novel fMRI-based visual decoding framework

Abstract

Objective. Visual perception decoding plays an important role in understanding our visual systems. Recent functional magnetic resonance imaging (fMRI) studies have made great advances in predicting the visual content of the single stimulus from the evoked response. In this work, we proposed a novel framework to extend previous works by simultaneously decoding the temporal and category information of visual stimuli from fMRI activities. Approach. 3 T fMRI data of five volunteers were acquired while they were viewing five categories of natural images with random presentation intervals. For each subject, we trained two classification-based decoding modules that were used to identify the occurrence time and semantic categories of the visual stimuli. In each module, we adopted recurrent neural network (RNN), which has proven to be highly effective for learning nonlinear representations from sequential data, for the analysis of the temporal dynamics of fMRI activity patterns. Finally, we integrated the two modules into a complete framework. Main results. The proposed framework shows promising decoding performance. The average decoding accuracy across five subjects was over 19 times the chance level. Moreover, we compared the decoding performance of the early visual cortex (eVC) and the high-level visual cortex (hVC). The comparison results indicated that both eVC and hVC participated in processing visual stimuli, but the semantic information of the visual stimuli was mainly represented in hVC. Significance. The proposed framework advances the decoding of visual experiences and facilitates a better understanding of our visual functions.

From Sight to Insight: A Multi-task Approach with the Visual Language Decoding Model

“…Individual higher-order visual areas exhibited higher accuracy while lower-level visual cortical areas displayed relatively lower accuracy. These outcomes align with previous research supporting the viewpoint that higher-level visual areas often demonstrate stronger classification decoding capabilities 15,30 . Concerning combined visual regions, the LVC region demonstrated higher classification accuracy than any of V1, V2, and V3, while HVC displayed higher accuracy than any of its constituent regions.…”

A dual‐channel language decoding from brain activity with progressive transfer training

“…All subjects were provided written informed consent before the MRI experiments, and protocols were approved by the Institutional Review Board of the Institute of Biophysics, Chinese Academy of Sciences. A detailed experimental description is shown in our previous work (Huang, Yan, Wang, Li, Yang, et al, 2020; Huang, Yan, Wang, Li, Zuo, et al, 2020; Huang, Yan, Wang, Yang, et al, 2021; C. Wang et al, 2020).…”

Using Deep Learning for Visual Decoding and Reconstruction from Brain Activity: A Review

“…More emphasis can be made on the decoding architecture to include temporal and categorical information from fMRI activity. A recent paper investigated simultaneously extracting both types of information from fMRI activity and shows higher accuracy from the decoding framework [11]. Improving the decoding and reconstruction to accurately use vital features to reconstruct images will be the future for brain decoding.…”

“…The goal of visual reconstruction is to map brain activity to decoded features and to use these decoded features to reconstruct seen stimuli [11]. This is an important field of computational neuroscience to not only have a better understanding of how the brain processes information but how it perceives information or objects.…”