CNN-based Encoding and Decoding of Visual Object Recognition in Space and Time

Abstract

Deep convolutional neural networks (CNNs) have been put forward as neurobiologically plausible models of the visual hierarchy. Using functional magnetic resonance imaging, CNN representations of visual stimuli have previously been shown to correspond to processing stages in the ventral and dorsal streams of the visual system. Whether this correspondence between models and brain signals also holds for activity acquired at high temporal resolution has been explored less exhaustively. Here, we addressed this question by combining CNNbased encoding models with magnetoencephalography (MEG). Human participants passively viewed 1000 images of objects while MEG signals were acquired. We modelled their high temporal resolution source-reconstructed cortical activity with CNNs, and observed a feedforward sweep across the visual hierarchy between 75-200 ms after stimulus onset. This spatiotemporal cascade was captured by the network layer representations, where the increasingly abstract stimulus representation in the hierarchical network model was reflected in different parts of the visual cortex, following the visual ventral stream. We further validated the accuracy of our encoding model by decoding stimulus identity in a left-out validation set of viewed objects, achieving state-of-the-art decoding accuracy.

Deep Neural Networks in Computational Neuroscience

“…In line with results from macaque IT, DNNs were furthermore able to explain within-category neural similarities, despite being trained on a categorization task that aims at abstracting away from differences across categoryexemplars (Khaligh-Razavi & Kriegeskorte, 2014). At a lower spatial, but higher temporal resolution, DNNs have also been shown to be predictive of visually evoked magnetoencephalography (MEG) data Cichy, Khosla, Pantazis, Torralba, & Oliva, 2016;Fritsche, G, Schoffelen, Bosch, & Gerven, 2017). On the behavioural level, deep networks exhibit similar behaviour to humans (Hong, Yamins, Majaj, & DiCarlo, 2016;Kheradpisheh, Ghodrati, Ganjtabesh, & Masquelier, 2016b, 2016aKubilius, Bracci, & Op de Beeck, 2016;Majaj, Hong, Solomon, & DiCarlo, 2015) and are currently the best-performing model in explaining human eye-movements in free viewing paradigms (Kümmerer, Theis, & Bethge, 2014).…”

Activations of Deep Convolutional Neural Network are Aligned with Gamma Band Activity of Human Visual Cortex

“…Previous work has established a correspondence between hierarchy of the DCNN and the fMRI responses measured across the human visual areas (Güçlü and van Gerven, 2015; Eickenberg et al, 2016; Seibert et al, 2016; Cichy et al, 2016b). Further research has shown that the activity of the DCNN matches the biological neural hierarchy in time as well (Cichy et al, 2016b; Seeliger et al, 2017). Studying intracranial recordings allowed us to extend previous findings by assessing the alignment between the DCNN and cortical signals at different frequency bands.…”

“…frequencies). With time-resolved magnetoencephalography (MEG) recordings it has been demonstrated that the correspondence between the DCNN and neural signals peaks in the first 200 ms (Cichy et al, 2016b; Seeliger et al, 2017). Here we test the remaining dimension: that biological visual object recognition is also specific to certain frequencies.…”

Multivariate pattern analysis for MEG: a comprehensive comparison of dissimilarity measures

“…RSA characterizes the similarity of measured responses to experimental conditions in representational dissimilarity matrices (RDMs). As RDMs can in principle be computed for any measurement modality, RSA on MEG opens the way to quantitatively relate rapidly emerging brain dynamics to other data, such as fMRI (Cichy et al, 2016b(Cichy et al, , 2013 in order to localize responses; computational models (Cichy et al, 2017a(Cichy et al, , 2016aKietzmann et al, 2017;Pantazis et al, 2017;Seeliger et al, 2017;Su et al, 2012;Wardle et al, 2016) in order to understand the underlying algorithms and representational format; to behaviour (Cichy et al, 2017b;Furl et al, 2017;Mur et al, 2013); and across species (Cichy et al, 2014).…”