The human visual system and CNNs can both support robust online translation tolerance following extreme displacements

Abstract

Visual translation tolerance refers to our capacity to recognize objects over a wide range of different retinal locations. Although translation is perhaps the simplest spatial transform that the visual system needs to cope with, the extent to which the human visual system can identify objects at previously unseen locations is unclear, with some studies reporting near complete invariance over 10 degrees and other reporting zero invariance at 4 degrees of visual angle. Similarly, there is confusion regarding the extent of translation tolerance in computational models of vision, as well as the degree of match between human and model performance. Here, we report a series of eye-tracking studies (total N = 70) demonstrating that novel objects trained at one retinal location can be recognized at high accuracy rates following translations up to 18 degrees. We also show that standard deep convolutional neural networks (DCNNs) support our findings when pretrained to classify another set of stimuli across a range of locations, or when a global average pooling (GAP) layer is added to produce larger receptive fields. Our findings provide a strong constraint for theories of human vision and help explain inconsistent findings previously reported with convolutional neural networks (CNNs).

Learning Online Visual Invariances for Novel Objects via Supervised and Self-Supervised Training

“…The distinction between trained and online invariance has been the focus of recent work on translation invariance (recognizing an object at novel retinal locations after having seen it at one location). Behavioral studies have highlighted the extent to which humans possess online translation invariance (Blything et al, 2020(Blything et al, , 2021, and it is often claimed that convolutional and pooling operations endow CNNs a similar degree of online invariance to translation (Marcus, 2018;LeCun and Bengio, 1995). However, several experiments have shown that this is not the case, and indeed, CNNs not only fail to support online invariance to translation, but also fail to support online invariance to scale, rotation, and flipping along the vertical and horizontal axis (Kauderer-Abrams, 2017;Gong et al, 2014;Blything et al, 2021;Chen et al, 2017).…”

“…Behavioral studies have highlighted the extent to which humans possess online translation invariance (Blything et al, 2020(Blything et al, , 2021, and it is often claimed that convolutional and pooling operations endow CNNs a similar degree of online invariance to translation (Marcus, 2018;LeCun and Bengio, 1995). However, several experiments have shown that this is not the case, and indeed, CNNs not only fail to support online invariance to translation, but also fail to support online invariance to scale, rotation, and flipping along the vertical and horizontal axis (Kauderer-Abrams, 2017;Gong et al, 2014;Blything et al, 2021;Chen et al, 2017). One approach to achieving online invariance in CNNs is through architectural modification, for example, adding a Global Average Pooling layer to the end of the convolutional block results in complete translation invariance (Blything et al, 2021).…”

“…However, several experiments have shown that this is not the case, and indeed, CNNs not only fail to support online invariance to translation, but also fail to support online invariance to scale, rotation, and flipping along the vertical and horizontal axis (Kauderer-Abrams, 2017;Gong et al, 2014;Blything et al, 2021;Chen et al, 2017). One approach to achieving online invariance in CNNs is through architectural modification, for example, adding a Global Average Pooling layer to the end of the convolutional block results in complete translation invariance (Blything et al, 2021). Additional architectural modifications have been introduced in order to support other types of transformation (e.g.…”

“…If it turns out that some training regimes allow standard CNNs to support online invariances, the introduction of additional architectural (innate) mechanisms may not be required either for engineering or psychological considerations. Biscione and Bowers (2020) and Blything et al (2021) first reported examples of a classic CNN model (VGG16), without any architectural changes, exhibiting complete online invariance to translation when the model was pretrained on the ImageNet dataset. That is, a network pretrained on Ima-geNet supported translation invariance to other, very different datasets (e.g.…”

“…Critically, this invariance is computed 'online': a transformed, unfamiliar object can often be recognized after a single presentation at a given pose. For example, after identifying a novel object that is projected at a given retinal location we can immediately identify the object across a wide range of retinal locations (Blything et al, 2020(Blything et al, , 2021. How the visual system succeeds under these conditions is still poorly understood, but we know that the this task is solved through hierarchical processing along the ventral stream that ends at the inferotemporal cortex, where the activation of neural populations are largely independent to object transformation (DiCarlo and Cox, 2007).…”