Recurrent Connections in the Primate Ventral Visual Stream Mediate a Trade-Off Between Task Performance and Network Size During Core Object Recognition

Abstract

The computational role of the abundant feedback connections in the ventral visual stream is unclear, enabling humans and nonhuman primates to effortlessly recognize objects across a multitude of viewing conditions. Prior studies have augmented feedforward convolutional neural networks (CNNs) with recurrent connections to study their role in visual processing; however, often these recurrent networks are optimized directly on neural data or the comparative metrics used are undefined for standard feedforward networks that lack these connections. In this work, we develop task-optimized convolutional recurrent (ConvRNN) network models that more correctly mimic the timing and gross neuroanatomy of the ventral pathway. Properly chosen intermediate-depth ConvRNN circuit architectures, which incorporate mechanisms of feedforward bypassing and recurrent gating, can achieve high performance on a core recognition task, comparable to that of much deeper feedforward networks. We then develop methods that allow us to compare both CNNs and ConvRNNs to finely grained measurements of primate categorization behavior and neural response trajectories across thousands of stimuli. We find that high-performing ConvRNNs provide a better match to these data than feedforward networks of any depth, predicting the precise timings at which each stimulus is behaviorally decoded from neural activation patterns. Moreover, these ConvRNN circuits consistently produce quantitatively accurate predictions of neural dynamics from V4 and IT across the entire stimulus presentation. In fact, we find that the highest-performing ConvRNNs, which best match neural and behavioral data, also achieve a strong Pareto trade-off between task performance and overall network size. Taken together, our results suggest the functional purpose of recurrence in the ventral pathway is to fit a high-performing network in cortex, attaining computational power through temporal rather than spatial complexity.

A temporal hierarchy of object processing in human visual cortex

“…The aligned, simultaneous representational changes in OTC and lower areas imply recurrence between them 36,37 and/or a common top-down input 38 . In either case, we’d expect shared trial-by-trial neural variability between areas 39 .…”

“…In principle, downstream neurons at a later stage of the visual hierarchy may build a coarse presentation at first based on their initial inputs, which may then help upstream neurons at an earlier hierarchical stage with smaller receptive fields disambiguate 50 what they see, and the disambiguated signals may, in turn, contribute to a refined downstream presentation, and so on. A similar mechanism with different sites of downstream and upstream neurons may underlie object recognition, and its further characterization likely requires fine-grained stimulus design combined with novel computational models 34 .…”

“…1D), and it occurred for an extended period and concurrently at all stages. Therefore, the temporal hierarchy we found may result from non-feedforward processes, including recurrence across stages [32][33][34] and a common top-down input 35 . In either case, we expect shared trialby-trial neural variability between areas 36 .…”

Efficient processing of natural scenes in visual cortex

“…Finally, the advent of deep learning may provide an interesting new approach to understanding the logic of neural circuits deeper in the brain, where the guiding principle is that circuits beyond the sensory periphery must self-organize through local learning rules to achieve whatever tasks are behaviorally necessary. Indeed, some authors have suggested that the hierarchy of visual cortical areas should be understood in analogy with the layers of a deep network (Yamins et al, 2014 ; Yamins and DiCarlo, 2016 ; Schrimpf et al, 2020 ; Muratore et al, 2022 ; Nayebi et al, 2022 ). Of course these circuits ultimately operate on inputs drawn from the natural world, and hence should adapt through learning to both scene statistics and the target task.…”

Training biologically plausible recurrent neural networks on cognitive tasks with long-term dependencies

“…First, the brain's connectivity is highly recurrent at multiple spatial scales, making recurrent neural networks (RNNs) [5] the modeling tool of choice in neuroscience [2,4,6]. The training and analysis of RNNs has advanced theory on the neural basis of perception [7][8][9], motor behavior [10][11][12][13], cognition [14][15][16][17][18][19][20][21][22][23][24][25][26][27], memory and navigation [28,29]. Second, physiological properties of neurons places a greater demand on the emergent computational properties of neural populations.…”