Low-level tuning biases in higher visual cortex reflect the semantic informativeness of visual features

Abstract

Representations of visual and semantic information can overlap in human visual cortex, with the same neural populations exhibiting sensitivity to low-level features (orientation, spatial frequency, retinotopic position), and high-level semantic categories (faces, scenes). It has been hypothesized that this relationship between low-level visual and high-level category neural selectivity reflects natural scene statistics, such that neurons in a given category-selective region are tuned for low-level features or spatial positions that are diagnostic of the region's preferred category. To address the generality of this "natural scene statistics" hypothesis, as well as how well it can account for responses to complex naturalistic images across visual cortex, we performed two complementary analyses. First, across a large set of rich natural scene images, we demonstrated reliable associations between low-level (Gabor) features and high-level semantic dimensions (indoor-outdoor, animacy, real-world size), with these relationships varying spatially across the visual field. Second, we used a large-scale fMRI dataset (the Natural Scenes Dataset) and a voxelwise forward encoding model to estimate the feature and spatial selectivity of neural populations throughout visual cortex. We found that voxels in category-selective visual regions exhibit systematic biases in their feature and spatial selectivity which are consistent with their hypothesized roles in category processing. We further showed that these low-level tuning biases are largely independent of viewed image category. Together, our results are consistent with a framework in which low-level feature selectivity contributes to the computation of high-level semantic category information in the brain.

Neural Selectivity for Real-World Object Size In Natural Images

“…The pRF estimates are then used to compute a single feature vector for each voxel and each image, by taking a weighted sum between the voxel’s pRF (a 2D Gaussian) and a stack of feature maps corresponding to the image of interest (see St-Yves and Naselaris (2017) for details on this approach). For the Gabor model, these maps are computed at 12 orientations and 8 spatial frequencies, for a total of 96 spatial feature maps (see Henderson et al (2022) for more details on model construction).…”

“…For the Gabor model, these maps are computed at 12 orientations and 8 spatial frequencies, for a total of 96 spatial feature maps (see Henderson et al (2022) for more details on model construction).…”

A texture statistics encoding model reveals hierarchical feature selectivity across human visual cortex

“…Similarly, when implementing the Gabor model, we also used 4 orientations and 4 spatial frequencies (0.36, 1.03, 2.97, and 8.57 cycles/deg). Each Gabor model feature was computed by filtering the image with two sinusoids that were 90° out of phase, squaring the output of these two filters, summing the two outputs, and taking the square root (see Henderson et al, 2022 for details on construction of a similar model). To extract features from the Alexnet model, we used the pretrained model weights for Alexnet available from the PyTorch model zoo, and we extracted activations from layers Conv1 and Conv2 (following the rectifying nonlinear activation function).…”

“…For both the Gabor and Alexnet models, we incorporated spatial pRF parameters into the construction of the model, similar to how the pRF was incorporated into the texture statistics model. Essentially, this procedure consists of extracting features in each pRF of the grid by taking a dot product of the relevant feature maps with each pRF (for similar approaches, see: Henderson et al, 2022; St-Yves and Naselaris, 2017). To simplify the fitting procedure, as well as making the models more comparable to the texture statistics encoding model, we used the same pRF parameters that had already been estimated using the texture statistics encoding model (see Methods: Model fitting procedure ).…”

“…To assess how the texture statistics model's ability to predict neural responses compares to that of other models, we implemented several commonly-used models of low-level visual features: a GIST model (Oliva and Torralba, 2001), a complex Gabor model (Henderson et al, 2022;St-Yves and Naselaris, 2017), and the first two layers of a pretrained Alexnet convolutional neural network model (Krizhevsky, 2014). The GIST model consists of spectral features (orientation and spatial frequency) that are coarsely localized in space, and was implemented using Matlab code provided by the authors of Oliva and Torralba, 2001. We evaluated the GIST model with two levels of spatial resolution: a 2x2 grid and a 4x4 grid.…”