A Critical Test of Deep Convolutional Neural Networks' Ability to Capture Recurrent Processing in the Brain Using Visual Masking

Abstract

Recurrent processing is a crucial feature in human visual processing supporting perceptual grouping, figure-ground segmentation, and recognition under challenging conditions. There is a clear need to incorporate recurrent processing in deep convolutional neural networks, but the computations underlying recurrent processing remain unclear. In this article, we tested a form of recurrence in deep residual networks (ResNets) to capture recurrent processing signals in the human brain. Although ResNets are feedforward networks, they approximate an excitatory additive form of recurrence. Essentially, this form of recurrence consists of repeating excitatory activations in response to a static stimulus. Here, we used ResNets of varying depths (reflecting varying levels of recurrent processing) to explain EEG activity within a visual masking paradigm. Sixty-two humans and 50 artificial agents (10 ResNet models of depths −4, 6, 10, 18, and 34) completed an object categorization task. We show that deeper networks explained more variance in brain activity compared with shallower networks. Furthermore, all ResNets captured differences in brain activity between unmasked and masked trials, with differences starting at ∼98 msec (from stimulus onset). These early differences indicated that EEG activity reflected “pure” feedforward signals only briefly (up to ∼98 msec). After ∼98 msec, deeper networks showed a significant increase in explained variance, which peaks at ∼200 msec, but only within unmasked trials, not masked trials. In summary, we provided clear evidence that excitatory additive recurrent processing in ResNets captures some of the recurrent processing in humans.

Human visual cortex and deep convolutional neural network care deeply about object background

“…This ability is especially crucial when the object and its background share similar features such as line orientations, curvatures and colors. Both humans and DCNNs showed enhanced performance when presented with pre-segmented objects compared to HUMAN VISUAL CORTEX AND DEEP CONVOLUTIONAL NEURAL NETWORKS CARE DEEPLY ABOUT OBJECT BACKGROUND 6 objects embedded in backgrounds (29)(30)(31). To investigate this further, we used images with identical target objects embedded in varying background complexities, allowing us to isolate human electroencephalography (EEG) recordings and DCNN activity related to target object categorical features versus object background.…”

Recurrent connectivity supports higher-level visual and semantic object representations in the brain

“…Traditional models argued that visual object recognition is underpinned by a largely feedforward process along the ventral visual pathway (DiCarlo et al, 2012; Isik et al, 2014; Kreiman & Serre, 2020; Serre et al, 2007), while inter-regional feedback is reserved for more complex situations such as occlusion or visually degraded images (Ganis et al, 2007; Kreiman & Serre, 2020; Loke et al, 2022; Rajaei et al, 2019; Wyatte et al, 2014), meaning it might not be initiated for unambiguous familiar images. However, it has also been argued that object recognition generally involves inter-regional recurrent connectivity (Bar et al, 2006; Clarke et al, 2011; Hegdé, 2008; Hochstein & Ahissar, 2002; Kar et al, 2019; Kietzmann et al, 2019; H. Schendan & Ganis, 2015; Spoerer et al, 2017; Wyatte et al, 2014).…”