Biological convolutions improve DNN robustness to noise and generalisation

Abstract

Deep Convolutional Neural Networks (DNNs) have achieved superhuman accuracy on standard image classification benchmarks. Their success has reignited significant interest in their use as models of the primate visual system, bolstered by claims of their architectural and representational similarities. However, closer scrutiny of these models suggests that they rely on various forms of shortcut learning to achieve their impressive performance, such as using texture rather than shape information. Such superficial solutions to image recognition have been shown to make DNNs brittle in the face of more challenging tests such as noise-perturbed or out-of-domain images, casting doubt on their similarity to their biological counterparts. In the present work, we demonstrate that adding fixed biological filter banks, in particular banks of Gabor filters, helps to constrain the networks to avoid reliance on shortcuts, making them develop more structured internal representations and more tolerant to noise. Importantly, they also gained around 20-30% improved accuracy when generalising to our novel out-of-domain test image sets over standard end-to-end trained architectures. We take these findings to suggest that these properties of the primate visual system should be incorporated into DNNs to make them more able to cope with real-world vision and better capture some of the more impressive aspects of human visual perception such as generalisation.

From photos to sketches - how humans and deep neural networks process objects across different levels of visual abstraction

“…Our findings reconcile the demonstrations of poor performance of CNNs on drawings ( Ballester & Araujo, 2016 ; Evans et al, 2021 ; Wang et al, 2019 ) with evidence for generalizable representations for natural images and drawings ( Fan et al, 2018 ). Our results furthermore suggest that processing of object images across levels of visual abstraction remains domain-general up to the penultimate pooling layer, after which a distinct representational format emerges that is more biased toward the domain of natural images and leads to incorrect classifications on drawings and sketches.…”

“…Recent studies investigating human-like generalization to drawings in CNNs have yielded conflicting results. On the one hand, there is evidence that CNNs trained on natural images cannot recognize abstract drawings and achieve poor classification performance with such images ( Ballester & Araujo, 2016 ; Evans, Malhotra, & Bowers, 2021 ; Wang, Ge, Xing, & Lipton, 2019 ). This might be attributed to texture bias in CNNs, referring to the recent demonstration that CNNs tend to rely more strongly on texture rather than shape information for object recognition, which contrasts with humans who rely more strongly on shape ( Geirhos, Janssen, Schütt, Rauber, Bethge, & Wichmann, 2019 ; Hermann, Chen, & Kornblith, 2020 ).…”

From photos to sketches - how humans and deep neural networks process objects across different levels of visual abstraction

“…Our findings reconcile the demonstrations of poor performance of CNNs on drawings (Ballester & Araujo, 2016;Evans et al, 2021;Wang et al, 2019) with evidence for generalizable representations for natural images and drawings (Fan et al, 2018). Our results furthermore suggest that processing of object images across levels of visual abstraction remains domain-general up to the penultimate pooling layer, after which a distinct representational format emerges that is more biased towards the domain of natural images and leads to incorrect classifications on drawings and sketches.…”

“…While the generalization capabilities in human recognition are still out of reach for networks with a feedforward architecture and supervised learning regime (Geirhos et al, 2019;Geirhos, Janssen, et al, 2018;Geirhos, Temme, et al, 2018), developing models that more closely match the human brain in terms of architecture (Evans et al, 2021;Kietzmann, Spoerer, et al, 2019) and learning rules (Zhuang et al, 2021) offer new perspectives for meeting this goal. Such models have yielded important insight in how the brain solves the general problem of object recognition and show improved generalization in some cases (Geirhos, Narayanappa, et al, 2020;Spoerer et al, 2017) but are still outmatched by humans (Geirhos, Meding, et al, 2020;Geirhos, Narayanappa, et al, 2020).…”

“…Recent studies investigating human-like generalization to drawings in CNNs have yielded conflicting results. On the one hand, there is evidence that CNNs trained on natural images cannot recognize abstract drawings and achieve poor classification performance with such images (Ballester & Araujo, 2016;Evans et al, 2021;Wang et al, 2019). This might be attributed to texture bias in CNNs, referring to the recent demonstration that CNNs tend to rely more strongly on texture rather than shape information for object recognition, which contrasts with humans who rely more strongly on shape (Geirhos et al, 2019;Hermann et al, 2020).…”