Depth in convolutional neural networks solves scene segmentation

Abstract

Feedforward deep convolutional neural networks (DCNNs) are, under specific conditions, matching and even surpassing human performance in object recognition in natural scenes.This performance suggests that the analysis of a loose collection of image features could support the recognition of natural object categories, without dedicated systems to solve specific visual subtasks. Research in humans however suggests that while feedforward activity may suffice for sparse scenes with isolated objects, additional visual operations ('routines') that aid the recognition process (e.g. segmentation or grouping) are needed for more complex scenes. Linking human visual processing to performance of DCNNs with increasing depth, we here explored if, how, and when object information is differentiated from the backgrounds they appear on. To this end, we controlled the information in both objects and backgrounds, as well as the relationship between them by adding noise, manipulating background congruence and systematically occluding parts of the image.Results indicate that with an increase in network depth, there is an increase in the distinction between object-and background information. For more shallow networks, results indicated a benefit of training on segmented objects. Overall, these results indicate that, de facto, scene segmentation can be performed by a network of sufficient depth. We conclude that the human brain could perform scene segmentation in the context of object identification without an explicit mechanism, by selecting or "binding" features that belong to the object and ignoring other features, in a manner similar to a very deep convolutional neural network.

Capsule networks as recurrent models of grouping and segmentation

“…First, although feedforward networks can in principle implement any function [54], recurrent networks can implement certain functions more efficiently. Flexible grouping and segmentation is exactly the kind of function that may benefit from recurrent computations (see also [55]). For example, to determine which local elements should be grouped into a global object, it helps to compute the global object first.…”

On the necessity of recurrent processing during object recognition: it depends on the need for scene segmentation

“…Recently, a multitude of studies have reconciled these seemingly inconsistent findings by indicating that recurrent processes might be employed adaptively, depending on the visual input: while feed-forward activity might suffice for simple scenes with isolated objects, more complex scenes or more challenging conditions (e.g. objects that are occluded or degraded), may need additional visual operations (‘routines’) requiring recurrent computations (Groen et al, 2018; Tang et al, 2018; Kar et al, 2019; Rajaei et al, 2019; Seijdel et al, 2020). For objects in isolation, or very simple scenes, rapid recognition may thus rely on a coarse and unsegmented feed-forward representation (Crouzet and Serre, 2011), while for more cluttered images recognition may require explicit encoding of spatial relationships between parts.…”

Leveraging spiking deep neural networks to understand the neural mechanisms underlying selective attention

“…Intriguingly, these networks not only parallel human performance on some object recognition tasks (VanRullen, 2017), but they also feature processing characteristics that bear a lot of resemblance to the visual ventral stream in primates (Eickenberg et al, 2017; Güçclü and van Gerven, 2015; Khaligh-Razavi and Kriegeskorte, 2014; Kubilius et al, 2018; Schrimpf et al, 2020; Yamins et al, 2014). Leveraging this link between neural processing and performance has already enhanced insight into the potential mechanisms underlying shape perception (Kubilius et al,2016), scene segmentation (Seijdel et al, 2020) and the role of recurrence during object recognition (Kar et al, 2019; Kietzmann et al, 2019b). DCNNs may thus provide a promising avenue for systematically investigating how different attention mechanisms may modulate neural processing and thereby, performance.…”