Shape-selective processing in deep networks: integrating the evidence on perceptual integration

Abstract

Understanding how deep neural networks resemble or differ from human vision becomes increasingly important with their widespread use in Computer Vision and as models in Neuroscience. A key aspect of human vision is shape: we decompose the visual world into distinct objects, use cues to infer their 3D geometries, and can group several object parts into a coherent whole. Do deep networks use the shape of objects similarly when they classify images? Research on this question has yielded conflicting results, with some studies showing evidence for shape selectivity in deep networks, while others demonstrated clear deficiencies. We argue that these conflicts arise from differences in experimental methods: whether studies use custom images in which only some features are available, images in which different features compete, image pairs that vary along different feature dimensions, or large sets of images to assess how representations vary overall. Each method offers a different, partial view of shape processing. After comparing their advantages and pitfalls, we propose two hypotheses that can reconcile previous results. Firstly, deep networks are sensitive to local, but not global shape. Secondly, the higher layers of deep networks discard some of the shape information that the lower layers are sensitive to. We test these hypotheses by comparing network representations for natural images and silhouettes in which local or global shape is degraded. The results support both hypotheses, but for different networks. Purely feed-forward convolutional networks are unable to integrate shape globally. In contrast, networks with residual or recurrent connections show a weak selectivity for global shape. This motivates further research into recurrent architectures for perceptual integration.

Teaching deep networks to see shape: Lessons from a simplified visual world

“…However, among the convolutional networks we tested the role of architecture was less clear. In the context of shape processing, one architectural component that is often discussed is lateral recurrent connectivity [19][20][21].…”

“…One key difference between human vision and deep networks is that humans strongly rely on shape when judging which category an object belongs to, whereas neural networks preferentially rely on texture [16,17] and similar surface-level properties [18]. In fact, networks seem to be largely unable to process the global shape of objects in a human-like manner [18][19][20].…”

“…These include explicit training on stylized images [17], stronger image augmentations [23], and adding custom network components [24]. However, increasing shape bias on cue conflict images may not automatically lead to human-like shape processing -it is only one out of several relevant metrics [20]. As a case in point, networks trained on stylized images to have a high shape bias seem to remain unable to process global object shape [19,21].…”

“…Evidence for a lack of shape processing in deep networks comes from several lines of research (see [20] for a review). While deep networks are able to use some shape information, e.g., to classify silhouettes, they are relatively unaffected by image distortions that disrupt global shape, such as scrambling of silhouette parts [16, 21].…”

“…This can be used to probe selectivity for shape and texture experimentally. However, neither of these techniques is perfect, since each alteration in shape also has an effect on textures and vice versa [20].…”