Recurrent Connections in the Primate Ventral Visual Stream Mediate a Tradeoff Between Task Performance and Network Size During Core Object Recognition

Abstract

The ventral visual stream (VVS) is a hierarchically connected series of cortical areas known to underlie core object recognition behaviors, enabling humans and non-human primates to effortlessly recognize objects across a multitude of viewing conditions. While recent feedforward convolutional neural networks (CNNs) provide quantitatively accurate predictions of temporally-averaged neural responses throughout the ventral pathway, they lack two ubiquitous neuroanatomical features: local recurrence within cortical areas and long-range feedback from downstream areas to upstream areas. As a result, such models are unable to account for the temporally-varying dynamical patterns thought to arise from recurrent visual circuits, nor can they provide insight into the behavioral goals that these recurrent circuits might help support. In this work, we augment CNNs with local recurrence and long-range feedback, developing convolutional RNN (ConvRNN) network models that more correctly mimic the gross neuroanatomy of the ventral pathway. Moreover, when the form of the recurrent circuit is chosen properly, ConvRNNs with comparatively small numbers of layers can achieve high performance on a core recognition task, comparable to that of much deeper feedforward networks. We then compared these models to temporally fine-grained neural and behavioral recordings from primates to thousands of images. We found that ConvRNNs better matched these data than alternative models, including the deepest feedforward networks, on two metrics: 1) neural dynamics in V4 and inferotemporal (IT) cortex at late timepoints after stimulus onset, and 2) the varying times at which object identity can be decoded from IT, including more challenging images that take longer to decode. Moreover, these results differentiate within the class of ConvRNNs, suggesting that there are strong functional constraints on the recurrent connectivity needed to match these phenomena. Finally, we find that recurrent circuits that attain high task performance while having a smaller network size as measured by number of units, rather than another metric such as the number of parameters, are overall most consistent with these data. Taken together, our results evince the role of recurrence and feedback in the ventral pathway to reliably perform core object recognition while subject to a strong total network size constraint.

Increasing neural network robustness improves match to macaque V1 eigenspectrum, spatial frequency preference and predictivity

“…The noise-corrected predictivity metric for a neuron was defined to be the Pearson’s correlation between the neuron’s response predictions and the observed neural responses divided by the square-root of the Spearman-Brown corrected cross-trial correlation of the neuron’s responses, consistent with prior work [14, 49].…”

Mouse visual cortex as a limited resource system that self-learns an ecologically-general representation

“…However, as observed by Harris et al (2019), there are many feedback connections from higher visual areas to lower visual areas. Incorporating these architectural motifs into our models and training these models using dynamic inputs may be useful for modeling temporal dynamics in mouse visual cortex, as has been recently done in primates (Nayebi et al, 2018;Kubilius et al, 2019;Nayebi et al, 2021). In addition, we can probe the functionality of, and test hypotheses for the utility of these feedback connections between visual areas using models that incorporate these features.…”

“…However, how can this be done generally for populations in which one does not have a prior characterization of what each cell encodes, especially those in higher visual areas? We took inspiration from methods that have proven useful in modeling primate and human visual, auditory, and motor cortex Kell et al, 2018;Michaels et al, 2020;Nayebi et al, 2021). Specifically, we aimed to identify the best performing class of similarity transforms needed to map the firing patterns of one animal's neural population to that of another (Figure 1A).…”

“…Before developing new models, we started by examining the common ImageNet models (VGG16 and ResNet-18) that have been benchmarked before on neural recordings from the primate ventral stream (Schrimpf et al, 2018;Zhuang et al, 2021;Nayebi et al, 2021) and from the mouse visual system (Cadena et al, 2019b;Shi et al, 2019;Conwell et al, 2020). Cadena et al (2019b) specifically found that training these architectures on the ImageNet categorization task did not always yield better (and in some cases worse) neural predictivity than their untrained counterparts.…”

A connectivity-constrained computational account of topographic organization in primate high-level visual cortex

“…This is likely because increasing the depth of the hierarchy of fully-connected layers also does not increase performance, and these layers with lateral connections can be seen as temporally unrolled versions of a deeper hierarchy. However, lateral connections have been demonstrated to provide additional functional power to convolutional layers (Nayebi et al, 2018;Kubilius et al, 2018), where increasing depth does typically come with superior performance. Further, other tasks such as segmentation may place greater demands on lateral connections (Linsley et al, 2018(Linsley et al, , 2020.…”