Generating the features for category representation using a deep convolutional neural network

Yu, Chen-Ping; Maxfield, Justin; Zelinsky, Gregory J.

doi:10.1167/16.12.251

Cited by 3 publications

(4 citation statements)

References 0 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Further, while the current fit between human brains and DNNs is stunning, it might be further improved, e.g. by increasing architectural similarity [80]. In particular, one possibility is to build a fovea-periphery organization as present in human retina into the DNN and to evaluate its consequences [81].…”

Section: Concrete Roadmap For Further Researchmentioning

confidence: 99%

Resolving the neural dynamics of visual and auditory scene processing in the human brain: a methodological approach

Cichy

Teng

2017

Phil. Trans. R. Soc. B

View full text Add to dashboard Cite

In natural environments, visual and auditory stimulation elicit responses across a large set of brain regions in a fraction of a second, yielding representations of the multimodal scene and its properties. The rapid and complex neural dynamics underlying visual and auditory information processing pose major challenges to human cognitive neuroscience. Brain signals measured non-invasively are inherently noisy, the format of neural representations is unknown, and transformations between representations are complex and often nonlinear. Further, no single non-invasive brain measurement technique provides a spatio-temporally integrated view. In this opinion piece, we argue that progress can be made by a concerted effort based on three pillars of recent methodological development: (i) sensitive analysis techniques such as decoding and cross-classification, (ii) complex computational modelling using models such as deep neural networks, and (iii) integration across imaging methods (magnetoencephalography/electroencephalography, functional magnetic resonance imaging) and models, e.g. using representational similarity analysis. We showcase two recent efforts that have been undertaken in this spirit and provide novel results about visual and auditory scene analysis. Finally, we discuss the limits of this perspective and sketch a concrete roadmap for future research.This article is part of the themed issue ‘Auditory and visual scene analysis’.

show abstract

Section: Concrete Roadmap For Further Researchmentioning

confidence: 99%

Resolving the neural dynamics of visual and auditory scene processing in the human brain: a methodological approach

Cichy

Teng

2017

Phil. Trans. R. Soc. B

View full text Add to dashboard Cite

show abstract

“…Second, practically speaking, the number of images we can present in a neuroimaging study will typically be fewer than ∼ 10 3 ; limited by such numbers, it is difficult to imaging a large advantage for more sophisticated networks. Given the evidence we found of non-feedforward processing in visual cortex, we hold that it would be more profitable, in future work, to develop and use nonfeedforward neural networks, relying on feedback isomorphic to the connectivity structure in the brain (Yu et al, 2016). In this way, we can compare the dynamics of a high-performing network with spatio-temporal neural activity in the brain so as to better understand the information flow.…”

Section: Confounding Factors In Data-driven Experimentsmentioning

confidence: 97%

Exploring spatio-temporal neural dynamics of the human visual cortex

Yang

Tarr

Aminoff

2018

Preprint

View full text Add to dashboard Cite

The human visual cortex is organized in a hierarchical manner. Although a significant body of evidence has been accumulated in support of this hypothesis, specific details regarding the spatial and temporal information flow remain open. Here we present detailed spatio-temporal correlation profiles of neural activity with low-level and high-level features derived from a "deep" (8-layer) neural network pre-trained for object recognition. These correlation profiles indicate an early-to-late shift from lowlevel features to high-level features and from low-level regions to higher-level regions along the visual hierarchy, consistent with feedforward information flow. To refine our understanding of information flow, we computed three sets of features from the low-and high-level features provided by the neural network: object-category-relevant low-level features (the common components between low-level and high-level features), low-level features roughly orthogonal to high-level features (the residual Layer 1 features), and unique high-level features that were roughly orthogonal to low-level features (the residual Layer 7 features). Contrasting the correlation effects of the common components and the residual Layer 1 features, we observed that the early visual cortex exhibits a similar amount of correlation with the two feature sets early in time (60 to 120 ms), but in a later time window, the early visual cortex exhibits a higher and longer correlation effect with the common components/low-level task-relevant features as compared to the low-level residual features-an effect unlikely to arise from purely feedforward information flow. Overall, our results indicate that non-feedforward processes, for example, top-down influences from mental representations of categories, may facilitate differentiation between these two types of low-level features within the early visual cortex. Common components between the low-level and high-level features, or the subspace of low-level featuresthat are relevant to high-level features.2. Residuals of low-level features. These are the low-level features that are roughly orthogonal to high-level features, or the features remaining after regressing out the common components.3. Residuals of high-level features. These are the high-level features that are roughly orthogonal to lowlevel features, or the features remaining after regressing out the common components.

show abstract

“…In particular, such non-feedforward networks should rely on feedback isomorphic to the connectivity structure measured in the primate brain, as well as employing smaller-scale recurrent structures such as lateral inhibition and considering the time scale of information flow within the network (e.g., Nayebi et al, 2018;Yu, Maxfield, & Zelinsky, 2016). In particular, such non-feedforward networks should rely on feedback isomorphic to the connectivity structure measured in the primate brain, as well as employing smaller-scale recurrent structures such as lateral inhibition and considering the time scale of information flow within the network (e.g., Nayebi et al, 2018;Yu, Maxfield, & Zelinsky, 2016).…”

Section: Choice Of Using Alexnetmentioning

confidence: 99%

“…Although it will be interesting to use features from these new feedforward networks to explain neural data, we hold that it would be more profitable to develop and use non-feedforward neural networks in future work, given the evidence we found of non-feedforward processing in the visual cortex. In particular, such non-feedforward networks should rely on feedback isomorphic to the connectivity structure measured in the primate brain, as well as employing smaller-scale recurrent structures such as lateral inhibition and considering the time scale of information flow within the network (e.g., Nayebi et al, 2018;Yu, Maxfield, & Zelinsky, 2016). In this way, we can compare the dynamics of a high-performing network with spatiotemporal neural activity in the brain so as to better understand the information flow.…”

Section: Choice Of Using Alexnetmentioning

confidence: 99%

Exploring spatiotemporal neural dynamics of the human visual cortex

Yang

Tarr

Aminoff

2019

Human Brain Mapping

View full text Add to dashboard Cite

The human visual cortex is organized in a hierarchical manner. Although previous evidence supporting this hypothesis has been accumulated, specific details regarding the spatiotemporal information flow remain open. Here we present detailed spatiotemporal correlation profiles of neural activity with low‐level and high‐level features derived from an eight‐layer neural network pretrained for object recognition. These correlation profiles indicate an early‐to‐late shift from low‐level features to high‐level features and from low‐level regions to higher‐level regions along the visual hierarchy, consistent with feedforward information flow. Additionally, we computed three sets of features from the low‐ and high‐level features provided by the neural network: object‐category‐relevant low‐level features (the common components between low‐level and high‐level features), low‐level features roughly orthogonal to high‐level features (the residual Layer 1 features), and unique high‐level features that were roughly orthogonal to low‐level features (the residual Layer 7 features). Contrasting the correlation effects of the common components and the residual Layer 1 features, we observed that the early visual cortex (EVC) exhibited a similar amount of correlation with the two feature sets early in time, but in a later time window, the EVC exhibited a higher and longer correlation effect with the common components (i.e., the low‐level object‐category‐relevant features) than with the low‐level residual features—an effect unlikely to arise from purely feedforward information flow. Overall, our results indicate that non‐feedforward processes, for example, top‐down influences from mental representations of categories, may facilitate differentiation between these two types of low‐level features within the EVC.

show abstract

Generating the features for category representation using a deep convolutional neural network

Cited by 3 publications

References 0 publications

Resolving the neural dynamics of visual and auditory scene processing in the human brain: a methodological approach

Resolving the neural dynamics of visual and auditory scene processing in the human brain: a methodological approach

Exploring spatio-temporal neural dynamics of the human visual cortex

Exploring spatiotemporal neural dynamics of the human visual cortex

Contact Info

Product

Resources

About