Central and peripheral vision for scene recognition: A neurocomputational modeling exploration

Abstract

What are the roles of central and peripheral vision in human scene recognition? Larson and Loschky (2009) showed that peripheral vision contributes more than central vision in obtaining maximum scene recognition accuracy. However, central vision is more efficient for scene recognition than peripheral, based on the amount of visual area needed for accurate recognition. In this study, we model and explain the results of Larson and Loschky (2009) using a neurocomputational modeling approach. We show that the advantage of peripheral vision in scene recognition, as well as the efficiency advantage for central vision, can be replicated using state-of-the-art deep neural network models. In addition, we propose and provide support for the hypothesis that the peripheral advantage comes from the inherent usefulness of peripheral features. This result is consistent with data presented by Thibaut, Tran, Szaffarczyk, and Boucart (2014), who showed that patients with central vision loss can still categorize natural scenes efficiently. Furthermore, by using a deep mixture-of-experts model ("The Deep Model," or TDM) that receives central and peripheral visual information on separate channels simultaneously, we show that the peripheral advantage emerges naturally in the learning process: When trained to categorize scenes, the model weights the peripheral pathway more than the central pathway. As we have seen in our previous modeling work, learning creates a transform that spreads different scene categories into different regions in representational space. Finally, we visualize the features for the two pathways, and find that different preferences for scene categories emerge for the two pathways during the training process.

Contextual Integration in Cortical and Convolutional Neural Networks

“…The field of deep learning has traditionally focused on feedforward models of visual processing. These models have been used to describe neural responses in the ventral stream of humans and other primates (Cadieu et al, 2014;Güçlü and van Gerven, 2015;Yamins and DiCarlo, 2016;Wang and Cottrell, 2017) and have resulted in many practical successes (Gu et al, 2017). More recently, convolutional neural networks that include recurrent connections (both lateral and top-down) have also been proposed (Spoerer et al, 2017).…”

Predicting the Driver's Focus of Attention: The DR(eye)VE Project

“…Also, half of these videoclips contain sub-sequences that were previously annotated as acting. To approximate as realistically as possible the visual field of attention of the driver, sampled videoclips are pre-processed following the procedure in [71]. As in [71] we leverage the Space Variant Imaging Toolbox [48] to implement this phase, setting the parameter that halves the spatial resolution every 2.3 • to mirror human vision [36], [71].…”

“…The resulting videoclip preserves details near to the fixation points in each frame, whereas the rest of the scene gets more and more blurred getting farther from fixations until only low-frequency contextual information survive. Coherently with [71] we refer to this process as foveation (in analogy with human foveal vision). Thus, pre-processed videoclips will be called foveated videoclips from now on.…”

The contributions of central and peripheral vision to scene-gist recognition with a 180° visual field

“…Most theories or models of scene-gist recognition are relatively silent on this issue Fei-Fei, VanRullen, Koch, & Perona, 2005;Oliva, 2005;. Thus, a thorough understanding of scene-gist recognition and computational models of rapid scene categorization must take into account differences in visual processing between central and peripheral vision (Wang & Cottrell, 2017).…”