High-Level Visual Encoding Model Framework with Hierarchical Ventral Stream-Optimized Neural Networks

Abstract

Visual encoding models based on deep neural networks (DNN) show good performance in predicting brain activity in low-level visual areas. However, due to the amount of neural data limitation, DNN-based visual encoding models are difficult to fit for high-level visual areas, resulting in insufficient encoding performance. The ventral stream suggests that higher visual areas receive information from lower visual areas, which is not fully reflected in the current encoding models. In the present study, we propose a novel visual encoding model framework which uses the hierarchy of representations in the ventral stream to improve the model’s performance in high-level visual areas. Under the framework, we propose two categories of hierarchical encoding models from the voxel and the feature perspectives to realize the hierarchical representations. From the voxel perspective, we first constructed an encoding model for the low-level visual area (V1 or V2) and extracted the voxel space predicted by the model. Then we use the extracted voxel space of the low-level visual area to predict the voxel space of the high-level visual area (V4 or LO) via constructing a voxel-to-voxel model. From the feature perspective, the feature space of the first model is extracted to predict the voxel space of the high-level visual area. The experimental results show that two categories of hierarchical encoding models effectively improve the encoding performance in V4 and LO. In addition, the proportion of the best-encoded voxels for different models in V4 and LO show that our proposed models have obvious advantages in prediction accuracy. We find that the hierarchy of representations in the ventral stream has a positive effect on improving the performance of the existing model in high-level visual areas.

A Convolutional Neural Network Interpretable Framework for Human Ventral Visual Pathway Representation

“…This creates a further appetite for model interpretability. The goal-driven approach is to characterize voxel responses through the feature space trained on high-level tasks, and the data-driven approach is to directly train the model with fMRI data to characterize voxel responses (Cadena et al 2019;Xiao et al 2022). It is essential to note that biological visual learning is a process of differentiation, wherein the learning involves discerning differences in existing visual features present in visual inputs rather than constructing new features for each new category (Konkle and Alvarez 2022).…”