Finding and recognizing objects in natural scenes: complementary computations in the dorsal and ventral visual systems

Abstract

Searching for and recognizing objects in complex natural scenes is implemented by multiple saccades until the eyes reach within the reduced receptive field sizes of inferior temporal cortex (IT) neurons. We analyze and model how the dorsal and ventral visual streams both contribute to this. Saliency detection in the dorsal visual system including area LIP is modeled by graph-based visual saliency, and allows the eyes to fixate potential objects within several degrees. Visual information at the fixated location subtending approximately 9° corresponding to the receptive fields of IT neurons is then passed through a four layer hierarchical model of the ventral cortical visual system, VisNet. We show that VisNet can be trained using a synaptic modification rule with a short-term memory trace of recent neuronal activity to capture both the required view and translation invariances to allow in the model approximately 90% correct object recognition for 4 objects shown in any view across a range of 135° anywhere in a scene. The model was able to generalize correctly within the four trained views and the 25 trained translations. This approach analyses the principles by which complementary computations in the dorsal and ventral visual cortical streams enable objects to be located and recognized in complex natural scenes.

Spatial representations in the primate hippocampus, and their functions in memory and navigation

“…In the context of the evolutionary development of the primate cortex, which include hierarchical organization, the development of a fovea, and mechanisms for object recognition for what is at the fovea and for moving the fovea to fixate objects (Rolls, 2016a;Rolls and Webb, 2014), we describe some of the cortical connections of the primate hippocampus. These connections provide a foundation for understanding the neurons recorded in the primate including human hippocampus.…”

Spatial coordinate transforms linking the allocentric hippocampal and egocentric parietal primate brain systems for memory, action in space, and navigation

“…However, the results described here show that even if that were the case, then temporal trace learning would be very helpful in allowing neurons to achieve useful and accurate invariance. However, the concept in VisNet is that temporal trace learning is sufficient to learn transform invariant representations of objects in the ventral visual system, with the great advantage that this mechanism can also account for other forms of invariance, including view‐invariant representations, which cannot be learned by a spatial coordinate transform (as different views of a given object may be completely different), but which VisNet with its temporal short‐term memory trace approach learns well (Bart & Hegde, ; Perry, Rolls, & Stringer, ; Robinson & Rolls, ; Rolls, , ; Rolls & Mills, ; Rolls & Stringer, ; Rolls & Webb, ; Wallis & Rolls, ; Webb & Rolls, ; Zhao et al, ).…”

The Visual Neuroscience of Robotic Grasping

“…Sabatini et al 2001;McLaughlin et al 2003;Lourens and Barakova 2007), or on increasingly invariant object representations for recognition as performed along the ventral stream (Ullman 1996; O'Reilly Fagg and Arbib (1998) and Munakata 2000; Cadieu et al 2007). The integration between the contributions of the two visual pathways is a subject almost unexplored (Rolls and Deco 2002;, and only a few studies explicitly deal with dorsal stream processing. The integration between the contributions of the two visual pathways is a subject almost unexplored (Rolls and Deco 2002;, and only a few studies explicitly deal with dorsal stream processing.…”