Not-So-CLEVR: learning same–different relations strains feedforward neural networks

Abstract

The advent of deep learning has recently led to great successes in various engineering applications. As a prime example, convolutional neural networks, a type of feedforward neural network, now approach human accuracy on visual recognition tasks like image classification and face recognition. However, here we will show that feedforward neural networks struggle to learn abstract visual relations that are effortlessly recognized by non-human primates, birds, rodents and even insects. We systematically study the ability of feedforward neural networks to learn to recognize a variety of visual relations and demonstrate that same-different visual relations pose a particular strain on these networks. Networks fail to learn same-different visual relations when stimulus variability makes rote memorization difficult. Further, we show that learning same-different problems becomes trivial for a feedforward network that is fed with perceptually grouped stimuli. This demonstration and the comparative success of biological vision in learning visual relations suggests that feedback mechanisms such as attention, working memory and perceptual grouping may be the key components underlying human-level abstract visual reasoning.

Beyond the feedforward sweep: feedback computations in the visual cortex

“…The challenge broke the state of the art in computer vision in 2011 right before the deep learning era. Today, the challenge seems to remain significant for modern DCNNs as shown by several groups 93–95 …”

“…In particular, Kim et al 95 . found a clear dichotomy between visual reasoning tasks: while spatial relations appeared to be easily learnable by feedforward neural networks (DCNNs and their extensions), same−different relations appear to pose a particular strain on these networks (i.e., they require deeper architectures and significantly more training examples to be learned).…”

“…Today, the challenge seems to remain significant for modern DCNNs as shown by several groups. [93][94][95] In particular, Kim et al 95 found a clear dichotomy between visual reasoning tasks: while spatial relations appeared to be easily learnable by feedforward neural networks (DCNNs and their extensions), same−different relations appear to pose a particular strain on these networks (i.e., they require deeper architectures and significantly more training examples to be learned). Ultimately, even with one million samples available to train the networks for each of the problems, learning same-different visual relations posed a challenge for these architectures when stimulus variability made rote memorization difficult (although see Ref.…”

Differential involvement of EEG oscillatory components in sameness vs. spatial-relation visual reasoning tasks

“…However, despite the remarkable accuracy reached in these recognition tasks, the limitations of DCNs are becoming increasingly evident (see Serre, 2019 for a very recent review). Beyond image categorization tasks, DCNs appear to struggle to learn to solve relatively simple visual reasoning tasks otherwise trivial for the human brain [6,7]. A recent study (Kim et al, 2018) thoroughly investigated the ability of DCN architectures to learn to solve various visual reasoning tasks and found an apparent dichotomy between problems: on the one hand stand tasks that require judging the spatial relations between items (Spatial Relationship -SR); on the other, those that require comparing items (Same-Different -SD).…”

“…Beyond image categorization tasks, DCNs appear to struggle to learn to solve relatively simple visual reasoning tasks otherwise trivial for the human brain [6,7]. A recent study (Kim et al, 2018) thoroughly investigated the ability of DCN architectures to learn to solve various visual reasoning tasks and found an apparent dichotomy between problems: on the one hand stand tasks that require judging the spatial relations between items (Spatial Relationship -SR); on the other, those that require comparing items (Same-Different -SD). Importantly, Kim This prompts the question of how biological visual systems handle such tasks so efficiently.…”