Noise-trained deep neural networks effectively predict human vision and its neural responses to challenging images

Abstract

Deep neural networks (DNNs) for object classification have been argued to provide the most promising model of the visual system, accompanied by claims that they have attained or even surpassed human-level performance. Here, we evaluated whether DNNs provide a viable model of human vision when tested with challenging noisy images of objects, sometimes presented at the very limits of visibility. We show that popular state-of-the-art DNNs perform in a qualitatively different manner than humans—they are unusually susceptible to spatially uncorrelated white noise and less impaired by spatially correlated noise. We implemented a noise training procedure to determine whether noise-trained DNNs exhibit more robust responses that better match human behavioral and neural performance. We found that noise-trained DNNs provide a better qualitative match to human performance; moreover, they reliably predict human recognition thresholds on an image-by-image basis. Functional neuroimaging revealed that noise-trained DNNs provide a better correspondence to the pattern-specific neural representations found in both early visual areas and high-level object areas. A layer-specific analysis of the DNNs indicated that noise training led to broad-ranging modifications throughout the network, with greater benefits of noise robustness accruing in progressively higher layers. Our findings demonstrate that noise-trained DNNs provide a viable model to account for human behavioral and neural responses to objects in challenging noisy viewing conditions. Further, they suggest that robustness to noise may be acquired through a process of visual learning.

Noise increases the correspondence between artificial and human vision

“…This research extends previous work showing that DNNs are severely affected by various image corruptions and that the patterns of errors they make on such images do not mirror the mistakes that humans make [5]. The SSNR threshold for the noise-trained DNN reported by Jang and colleagues was slightly lower than that of the human viewers [3]. This is in line with previous work, which found that data augmentation can lead to superhuman performance on the specific image corruptions seen during training [6].…”

“…Rusak and colleagues [7] also demonstrated that careful noise training can help DNNs generalize to unseen image corruptions as well. The neuroimaging results presented by Jang and colleagues [3] provides novel evidence that noise training brings the network's internal information processing, not just its output, into greater alignment with that of the human visual system.…”

“…Human viewers were asked to "paint" the regions of the image that were most informative for their decision. Examples of these relevance maps can be seen in Fig 1 . Both recognition thresholds and relevance maps were more human-like for noise-robust DNNs [3].…”

“…They found that although early visual regions were well captured by the activity of early network layers, significant variance was left unaccounted for in high-level visual areas [11]. The comparison in Jang and colleagues found that noise training increased the brain-model correspondence particularlyAU : PerPLOSstyle; italicsshouldnotbeusedforemphasisofwords:Pleaseconfirmth at higher-level visual areas [3]. Thus, together, these results point to candidate model refinements to ultimately build better models of the neural information processing underlying human vision.…”

Development of object recognition

“…Viewpoint invariance is a computationally challenging process because viewpoint changes, particularly the rotation of an object in depth, can radically change the shape of the object on the retina (Marr & Nishihara, 1978). Indeed, despite the success of artificial neural networks (ANNs) in several domains of object recognition (Jang, McCormack, & Tong, 2021;Kriegeskorte, 2015), identifying objects from unfamiliar orientations remains difficult even for state-of-the-art models (Barbu et al, 2019). By contrast, human adults readily identify objects from novel orientations, even when the objects are unfamiliar (Biederman & Bar, 1999;Humphrey & Jolicoeur, 1993;Tarr & Bülthoff, 1995).…”