Crowding reveals fundamental differences in local vs. global processing in humans and machines

Abstract

Feedforward Convolutional Neural Networks (ffCNNs) have become state-of-the-art models both in computer vision and neuroscience. However, human-like performance of ffCNNs does not necessarily imply human-like computations. Previous studies have suggested that current ffCNNs do not make use of global shape information. However, it is currently unclear whether this reflects fundamental differences between ffCNN and human processing or is merely an artefact of how ffCNNs are trained. Here, we use visual crowding as a well-controlled, specific probe to test global shape computations. Our results provide evidence that ffCNNs cannot produce human-like global shape computations for principled architectural reasons. We lay out approaches that may address shortcomings of ffCNNs to provide better models of the human visual system. texture. This training dataset biased an ffCNN (ResNet50; He, Zhang, Ren, & Sun, 2016) towards shape-level features, because textural information was no longer useful for classifying this dataset. They validated the network's shape-bias by showing increased robustness to local noise and textural changes. Alternatively, ffCNNs may be incapable of matching human global computations for principled architectural reasons. Even though Geirhos et al.'s network was able to ignore local features, it may not use global computations in the same way as humans. One difficulty in addressing this question is that there is no consensus about how to experimentally diagnose how deep networks compute global information.

Capsule networks as recurrent models of grouping and segmentation

“…Hence, our results show that, given adequate priors, CapsNets explain uncrowding. We have shown that ffCNNs and CNNs with lateral or top-down recurrent connections do not produce uncrowding, even when they are trained identically on groups of identical shapes and successfully learn on the training data, comparably to the CapsNets (furthermore, we showed previously that ffCNNs trained on large datasets, which are often used as general models of vision, do not show uncrowding either; [ 17 ]). This shows that merely training networks on groups of identical shapes is not sufficient to explain uncrowding.…”

“…In previous work, we have shown that pretrained ffCNNs cannot explain uncrowding [ 17 ], even if they are biased towards global shape processing [ 13 ]. Currently, CapsNets cannot be trained on large-scale tasks such as ImageNet because routing by agreement is computationally too expensive.…”

“…An important point of discussion concerns global visual processing. It was suggested that ffCNNs mainly focus on local, texture-like features, while humans harness global shape computations ([ 9 , 13 – 17 ]; but see [ 18 ]). In this context, it was shown that changing local features of an object, such as its texture or edges, leads ffCNNs to misclassify [ 13 , 14 ], while humans can still easily classify the object based on its global shape.…”

“…Previously, we have shown that these global effects of crowding cannot be explained by models based on the classic framework of vision, including ffCNNs [ 9 , 17 , 38 ]. Here, we propose a new framework to understand these global effects.…”

Flexible contextual modulation of naturalistic texture perception in peripheral vision

“…In particular, much work with vernier and letter stimuli showed that even small changes to the contextual stimuli, or changes far away from the target, can lead to target-surround ungrouping and a considerable reduction in crowding (Kooi, Toet, Tripathy, & Levi, 1994; Manassi, Sayim, & Herzog, 2012; Manassi et al, 2016; Manassi, Hermens, Francis, & Herzog, 2015; Manassi et al, 2013; Saarela, Sayim, Westheimer, & Herzog, 2009), a phenomenon known as “uncrowding”. It has been argued that these results show a failure of feedforward pooling models, such as the SS model, and that this failure is due to their lack of recurrent processes of grouping and segmentation (Doerig et al, 2019; Doerig, Bornet, Choung, & Herzog, 2020; Herzog et al, 2015; Francis, Manassi, & Herzog, 2017). Furthermore, current SS model implementations also fail to capture the peripheral appearance of natural scenes that contain strong grouping and segmentation cues (Wallis et al, 2019).…”