Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence

Cichy, Radoslaw Martin; Khosla, Aditya; Pantazis, Dimitrios; Torralba, Antonio; Oliva, Aude

doi:10.1038/srep27755

Cited by 682 publications

(715 citation statements)

References 58 publications

Supporting

Mentioning

647

Contrasting

Order By: Relevance

“…A CNN is not designed to be biologically plausible, although we may draw a loose analogies between a CNN and the human cortex about their processing stages and representations (VanRullen, 2017). Comparative fMRI studies show that representations in the CNN correlate with emerging visual representations in the human brain (Cichy, Khosla, Pantazis, & Oliva, 2017;Cichy, Khosla, Pantazis, Torralba, & Oliva, 2016). Semantic similarity corresponds to more processed visual representations found in final CNN layers (in our case, fc7) or the inferotemporal cortex in the brain, whereas perceptual similarity is based on low-level representations in lower layers of a CNN or in the visual cortex.…”

Section: Discussionmentioning

confidence: 99%

Visual properties and memorising scenes: Effects of image-space sparseness and uniformity

Lukavský

Děchtěrenko

2017

Atten Percept Psychophys

View full text Add to dashboard Cite

Previous studies have demonstrated that humans have a remarkable capacity to memorise a large number of scenes. The research on memorability has shown that memory performance can be predicted by the content of an image. We explored how remembering an image is affected by the image properties within the context of the reference set, including the extent to which it is different from its neighbours (imagespace sparseness) and if it belongs to the same category as its neighbours (uniformity). We used a reference set of 2,048 scenes (64 categories), evaluated pairwise scene similarity using deep features from a pretrained convolutional neural network (CNN), and calculated the image-space sparseness and uniformity for each image. We ran three memory experiments, varying the memory workload with experiment length and colour/greyscale presentation. We measured the sensitivity and criterion value changes as a function of image-space sparseness and uniformity. Across all three experiments, we found separate effects of 1) sparseness on memory sensitivity, and 2) uniformity on the recognition criterion. People better remembered (and correctly rejected) images that were more separated from others. People tended to make more false alarms and fewer miss errors in images from categorically uniform portions of the image-space. We propose that both image-space properties affect human decisions when recognising images. Additionally, we found that colour presentation did not yield better memory performance over grayscale images.

show abstract

Section: Discussionmentioning

confidence: 99%

Visual properties and memorising scenes: Effects of image-space sparseness and uniformity

Lukavský

Děchtěrenko

2017

Atten Percept Psychophys

View full text Add to dashboard Cite

show abstract

“…These models, which in essence are a generalized form of principles first proposed by Hubel and Wiesel, have garnered increasing attention because they approach human levels of performance on tasks such as object recognition for real-world images [129,130] and show a degree of correspondence between individual layers of the networks and different levels of neural visual object processing [131–133]. CNNs that perform better on object-recognition tasks are found to be better predictors of neuronal spiking data in monkey inferotemporal cortex [134], suggesting the importance of a goal driven approach for creating a model of a given sensory system.…”

Section: Figurementioning

confidence: 99%

Making Sense of Real-World Scenes

Malcolm

Groen

Baker

2016

Trends in Cognitive Sciences

126

128

View full text Add to dashboard Cite

To interact with the world, we have to make sense of the continuous sensory input conveying information about our environment. A recent surge of studies has investigated the processes enabling scene understanding, using increasingly complex stimuli and sophisticated analyses to highlight the visual features and brain regions involved. However, there are two major challenges to producing a comprehensive framework for scene understanding. First, scene perception is highly dynamic, subserving multiple behavioral goals. Second, a multitude of different visual properties co-occur across scenes and may be correlated or independent. We synthesize the recent literature and argue that for a complete view of scene understanding, it is necessary to account for both differing observer goals and the contribution of diverse scene properties.

show abstract

“…MEG/EEG studies have furthermore shown that early layers of DNNs have a peak explained variance that is earlier than higher-tier DNN layers (Cichy, Khosla, Pantazis, Torralba, & Oliva, 2016;Ramakrishnan, Scholte, Smeulders, & Ghebreab, 2016). In addition, the DNN model has been shown to predict neural responses in IT, both from humans and macaque, much better than any other computational model (Khaligh-Razavi & Kriegeskorte, 2014;Yamins et al, 2014).…”

mentioning

confidence: 99%

Visual pathways from the perspective of cost functions and multi-task deep neural networks

Scholte

Losch

Ramakrishnan

et al. 2018

Cortex

View full text Add to dashboard Cite

Deep learning Cost functions RepresentationsVisual processing a b s t r a c t Vision research has been shaped by the seminal insight that we can understand the higher-tier visual cortex from the perspective of multiple functional pathways with different goals. In this paper, we try to give a computational account of the functional organization of this system by reasoning from the perspective of multi-task deep neural networks. Machine learning has shown that tasks become easier to solve when they are decomposed into subtasks with their own cost function. We hypothesize that the visual system optimizes multiple cost functions of unrelated tasks and this causes the emergence of a ventral pathway dedicated to vision for perception, and a dorsal pathway dedicated to vision for action.To evaluate the functional organization in multi-task deep neural networks, we propose a method that measures the contribution of a unit towards each task, applying it to two networks that have been trained on either two related or two unrelated tasks, using an identical stimulus set. Results show that the network trained on the unrelated tasks shows a decreasing degree of feature representation sharing towards higher-tier layers while the network trained on related tasks uniformly shows high degree of sharing.We conjecture that the method we propose can be used to analyze the anatomical and functional organization of the visual system and beyond. We predict that the degree to which tasks are related is a good descriptor of the degree to which they share downstream cortical-units.

show abstract

Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence

Cited by 682 publications

References 58 publications

Visual properties and memorising scenes: Effects of image-space sparseness and uniformity

Visual properties and memorising scenes: Effects of image-space sparseness and uniformity

Making Sense of Real-World Scenes

Visual pathways from the perspective of cost functions and multi-task deep neural networks

Contact Info

Product

Resources

About