Eight open questions in the computational modeling of higher sensory cortex

Abstract

Eight open questions in the computational modeling of higher sensory cortexPropelled by recent advances in biologically-inspired computer vision and artificial intelligence, the past five years have seen significant progress in using deep neural networks to model response patterns of neurons in higher visual cortical areas. In this paper, we briefly review this progress and then discuss eight key "open questions" that we believe will drive research in computational models of sensory systems over the next five years, both in visual cortex and beyond. Throughout, our focus is on challenges that will require both cutting-edge algorithmic developments as well as next-generation Propelled by advances in biologically-inspired computer vision and artificial intelligence, the past five years have seen significant progress in using deep neural networks to model response patterns of neurons in visual cortex. In this paper, we briefly review this progress and then discuss eight key "open questions" that we believe will drive research in computational models of sensory systems over the next five years, both in visual cortex and beyond.Any scientific development of long-term value opens up as many new questions as it answers. This is certainly the case with recent progress in building deep neural network models of visual cortex. In this piece, our goal is to briefly describe these recent advances, and to outline what we consider to be the most interesting open problems in cortical modeling, both in vision and beyond. Throughout, our focus is on questions that will require both cutting-edge algorithmic developments as well as next-generation neuroscience and cognitive science experiments. Brief Review of Recent Progress

Evidence that recurrent circuits are critical to the ventral stream’s execution of core object recognition behavior

“…In a single, natural viewing fixation (∼200 ms), primates can rapidly identify objects in the central visual field, despite various identity preserving image transformations, a behavior termed core object recognition (DiCarlo et al, 2012). Understanding the brain mechanisms that seamlessly solve this challenging computational problem has been a key goal of visual neuroscience (Riesenhuber and Poggio, 2000;Yamins and DiCarlo, 2016). Previous studies (Freiwald et al, 2009;Hung et al, 2005;Majaj et al, 2015) have shown that object categories and identities are explicitly represented in the pattern of neural activity in the primate inferior temporal (IT) cortex, and that specific IT neural population codes are sufficient to explain and predict primate core object recognition.…”

Fast Recurrent Processing via Ventrolateral Prefrontal Cortex Is Needed by the Primate Ventral Stream for Robust Core Visual Object Recognition

“…A goal of visual neuroscience is to identify and model the brain circuitry that seamlessly solves the challenging computational problem of rapid visual object categorization (DiCarlo and Cox, 2007;Riesenhuber and Poggio, 2000;Yamins and DiCarlo, 2016). Previous studies (Freiwald et al, 2009;Hung et al, 2005;Kar et al, 2019;Logothetis and Sheinberg, 1996;Majaj et al, 2015) show that the pattern of neural activity in primate inferior temporal (IT) cortex can explicitly represent visual object identities.…”

A connectivity-constrained computational account of topographic organization in primate high-level visual cortex

“…Extensive work in computational systems and cognitive neuroscience has demonstrated that task-driven computational models learn representations that better account for neural responses in visual cortex than models which are designed by hand (Yamins et al, 2014;Khaligh-Razavi and Kriegeskorte, 2014;Yamins and DiCarlo, 2016). More recent work has shown that a supervised version of task-driven learning is not essential to superior prediction of neural data, with semi-supervised contrastive learning algorithms performing very close to the supervised state-of-the-art (Zhuang et al, 2020).…”