Bio-inspired neural networks implement different recurrent visual processing strategies than task-trained ones do

Abstract

Behavioral studies suggest that recurrence in the visual system is important for processing degraded stimuli. There are two broad anatomical forms this recurrence can take, lateral or feedback, each with different assumed functions. Here we add four different kinds of recurrence — two of each anatomical form — to a feedforward convolutional neural network and find all forms capable of increasing the ability of the network to classify noisy digit images. Specifically, we take inspiration from findings in biology by adding predictive feedback and lateral surround suppression. To compare these forms of recurrence to anatomically-matched counterparts we also train feedback and lateral connections directly to classify degraded images. Counter-intuitively, we find that the anatomy of the recurrence is not related to its function: both forms of task-trained recurrence change neural activity and behavior similarly to each other and differently from their bio-inspired anatomical counterparts. By using several analysis tools frequently applied to neural data, we identified the distinct strategies used by the predictive versus task-trained networks. Specifically, predictive feedback de-noises the representation of noisy images at the first layer of the network and decreases its dimensionality, leading to an expected increase in classification performance. Surprisingly, in the task-trained networks, representations are not de-noised over time at the first layer (in fact, they become `noiser' and dimensionality increases) yet these dynamics do lead to de-noising at later layers. The analyses used here can be applied to real neural recordings to identify the strategies at play in the brain. Our analysis of an fMRI dataset weakly supports the predictive feedback model but points to a need for higher-resolution cross-regional data to understand recurrent visual processing.

Deep problems with neural network models of human vision

“…Recent studies investigated involuntary visual experiences using generative neural network models, such as in memory replay (van de Ven, Siegelmann, & Tolias, 2020), intrusive imagery (Cushing et al, 2023), and adversarial dreaming (Deperrois, Petrovici, Senn, & Jordan, 2022). Regarding voluntary visual mental imagery, some key strategies may involve modeling the retrieval process of representations pertaining to semantic information and visual features , and incorporating biologically inspired recurrence in visual imagery processing (Lindsay, Mrsic-Flogel, & Sahani, 2022).…”

Disentangling Mixed Classes of Covariability in Large-Scale Neural Data

“…Identifying invariances is crucial for reproducibility and interpretation, as non-unique solutions may prohibit clear comparison across datasets [Dyer et al, 2017, Gallego et al, 2020]. This issue is ever more important with the recent increase in popularity of comparisons of neural data to task-trained neural network models, whose representations are known to be sensitive to model specifications such as architecture and inputs [Lindsay et al, 2022, Williams et al, 2021]. Going forward, matrix and tensor decompositions could prove useful for comparing latent representations by virtue of their interpretability and tractability.…”

Generalizing biological surround suppression based on center surround similarity via deep neural network models

“…Our findings extend the ability of generic CNNs as models of visual cortices. The mismatches we found may inspire future studies of contextual effects in deep neural networks with more sophisticated circuitry, including the role of divisive normalization [47][48][49][50][51]64], recurrent connections and feedback [65][66][67][68][69][70] Dense 1000-Softmax Table 1. CNN architectures used in this study.…”