Atoms of recognition in human and computer vision

Abstract

Discovering the visual features and representations used by the brain to recognize objects is a central problem in the study of vision. Recently, neural network models of visual object recognition, including biological and deep network models, have shown remarkable progress and have begun to rival human performance in some challenging tasks. These models are trained on image examples and learn to extract features and representations and to use them for categorization. It remains unclear, however, whether the representations and learning processes discovered by current models are similar to those used by the human visual system. Here we show, by introducing and using minimal recognizable images, that the human visual system uses features and processes that are not used by current models and that are critical for recognition. We found by psychophysical studies that at the level of minimal recognizable images a minute change in the image can have a drastic effect on recognition, thus identifying features that are critical for the task. Simulations then showed that current models cannot explain this sensitivity to precise feature configurations and, more generally, do not learn to recognize minimal images at a human level. The role of the features shown here is revealed uniquely at the minimal level, where the contribution of each feature is essential. A full understanding of the learning and use of such features will extend our understanding of visual recognition and its cortical mechanisms and will enhance the capacity of computational models to learn from visual experience and to deal with recognition and detailed image interpretation.T he human visual system makes highly effective use of limited information (1, 2). As shown below ( Fig. 1 and Figs. S1 and S2), it can recognize consistently subconfigurations that are severely reduced in size or resolution. Effective recognition of reduced configurations is desirable for dealing with image variability: Images of a given category are highly variable, making recognition difficult, but this variability is reduced at the level of recognizable but minimal subconfigurations (Fig. 1B). Minimal recognizable configurations (MIRCs) are useful for effective recognition, but, as shown below, they also are computationally challenging because each MIRC is nonredundant and therefore requires the effective use of all available information. We use them here as sensitive tools to identify fundamental limitations of existing models of visual recognition and directions for essential extensions.A MIRC is defined as an image patch that can be reliably recognized by human observers and which is minimal in that further reduction in either size or resolution makes the patch unrecognizable (below criterion) (Methods). To discover MIRCs, we conducted a large-scale psychophysical experiment for classification. We started from 10 greyscale images, each showing an object from a different class (Fig. S3), and tested a large hierarchy of patches at different positions and decreasing size and res...

Deep Neural Networks in Computational Neuroscience

“…On the behavioural level, deep networks exhibit similar behaviour to humans (Hong, Yamins, Majaj, & DiCarlo, 2016;Kheradpisheh, Ghodrati, Ganjtabesh, & Masquelier, 2016b, 2016aKubilius, Bracci, & Op de Beeck, 2016;Majaj, Hong, Solomon, & DiCarlo, 2015) and are currently the best-performing model in explaining human eye-movements in free viewing paradigms (Kümmerer, Theis, & Bethge, 2014). Despite these advances, however, current DNNs still exhibit substantial differences in how they process and recognize visual stimuli (Linsley, Eberhardt, Sharma, Gupta, & Serre, 2017;Rajalingham et al, 2018;Ullman, Assif, Fetaya, & Harari, 2016), how they generalize to atypical category instances (Saleh, Elgammal, & Feldman, 2016), and how they perform under image manipulations, including reduced contrast and additive noise (Geirhos et al, 2017). Yet, the overall success clearly illustrates the power of DNN models for computational neuroscience.…”

Skeletal descriptions of shape provide unique perceptual information for object recognition

“…For instance, local contour information may be particularly useful for making subordinate-level category distinctions where the skeletons of objects are roughly the same 54,71 . Similarly, texture statistics and feature descriptions have been shown to be important indicators of both basic 11,72 and superordinate-level 73,74 object distinctions. And though it is currently unclear whether feedforward neural network models (such as the one tested here) incorporate global shape information 75,76 , more biologically plausible models with recurrent or generative architectures may begin to approximate human shape perception 68 .…”

Customer Analysis via Video Analytics: Customer Detection with Multiple Cues

“…An ensemble approach of Malisiewicz et al [27] predicts a class based on combined predictions from multiple linear Support Vector Machines (SVMs), each trained on only one example. Later development takes a deep learning approach [11][12] [13]. Although, most deep learning object detections do not require image features and can directly take image intensities as input.…”