Effective and Efficient ROI-wise Visual Encoding Using an End-to-End CNN Regression Model and Selective Optimization

Abstract

In neuroscience, visual encoding based on functional magnetic resonance imaging (fMRI) has been attracting much attention, especially with the recent development of deep learning. Visual encoding model is aimed at predicting subjects' brain activity in response to presented image stimuli . Current visual encoding models firstly extract image features through a pre-trained convolutional neural network (CNN) model, and secondly learn to linearly map the extracted CNN features to each voxel. However, it is hard for the two-step manner of visual encoding model to guarantee the extracted features are linearly well-matched with fMRI voxels, which reduces final encoding performance. Analogizing the development of the computer vision domain, we introduced the end-to-end manner into the visual encoding domain. In this study, we designed an end-to-end convolution regression model (ETECRM) and selective optimization based on the region of interest (ROI)-wise manner to accomplish more effective and efficient visual encoding. The model can automatically learn to extract better-matched features for encoding performance based on the end-to-end manner. The model can directly encode an entire visual ROI containing enormous voxels for encoding efficiency based on the ROI-wise manner, where the selective optimization was used to avoid the interference of some ineffective voxels in the same ROI. Experimental results demonstrated that ETECRM obtained improved encoding performance and efficiency than previous two-step models. Comparative analysis implied that the end-to-end manner and large volume of fMRI data are potential for the visual encoding domain.

BigGAN-based Bayesian Reconstruction of Natural Images from Human Brain Activity

“…Through training an encoding model to construct the mapping from visual stimuli space into fMRI voxel space, we can predict the fMRI voxels from Gallant-pattern images based on the pre-trained encoding model. In this study, we used the pre-trained encoding model for V1, V2, and V3 in our previous work (Qiao, et al 2019b) because the lower-level visual cortices are mainly responsible for the detailed visual information such as texture, location, and so on. Under the Bayesian framework, we evaluated the similarity between generated images and corresponding visual stimuli by measuring the distance between the true and predicted voxels.…”

“…Eventually, thousands of regression models are constructed for several visual ROIs, which is inefficient. In this study, we employed an effective and efficient encoding model as our previous study (Qiao, et al 2019b). As the high-level visual areas are difficult to encode and have bad encoding performance, we employed the encoding model for V1, V2, and V3 to preserve the fidelity of visual reconstruction.…”

“…To avoid the interfering of ineffective voxels during ROI-wise encoding, we used weighted correlation loss and noise regularization to accomplish selective optimization and update all of the parameters of convolution regression model. Further details of the encoding model were described in a previous study (Qiao, et al 2019b) and corresponding open-source repository (https://github.com/KaiQiao1992/ETECRM).…”

High-Level Visual Encoding Model Framework with Hierarchical Ventral Stream-Optimized Neural Networks

“…Data-driven visual encoding models can realize the nonlinear mapping from the visual stimuli to the brain response and learn the representation directly from the fMRI data in an end-to-end manner [ 26 , 27 , 28 , 29 , 30 , 31 , 32 ]. For example, Seeliger et al [ 31 ] proposed an end-to-end encoding model that simultaneously represents the neural information processing between different visual cortex and shows good fitting ability on the voxel activity of early visual areas.…”

Relating CNNs with brain: Challenges and findings

“…This nonlinear mapping replaces traditional linear mapping and is intended to improve brain activity prediction accuracy. Qiao et al [2021] proposed an end-to-end CNN regression model for visual encoding based on fMRI data. As a result, this model could develop appropriate feature representations and linear regression weights for visual cortical responses while also significantly improving prediction performance.…”