Predictive coding feedback results in perceived illusory contours in a recurrent neural network

Pang, Zhaoyang; O’May, Callum Biggs; Choksi, Bhavin; VanRullen, Rufin

doi:10.1016/j.neunet.2021.08.024

Cited by 38 publications

(28 citation statements)

References 41 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Augmenting a feedforward CNN with predictive coding recurrent dynamics allowed us to (i) analyse explicit classification decisions (square vs. inducers) and, unlike other related work [46,22,25], (ii) visualize reconstructed inputs from the model's viewpoint. As reported in a preliminary version of this study recently published in a conference workshop [52], we found that, compared to a feedforward baseline, the recurrent dynamics led the network to perceive more illusory contours. Notably, by inspecting the network's reconstructions, we were able to directly visualize the network's internal representation of the stimulus, which provides a much clearer measure of "illusory perception" than previous works.…”

Section: Discussionsupporting

confidence: 72%

Predictive coding feedback results in perceived illusory contours in a recurrent neural network

Pang

O’May

Choksi

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Modern feedforward convolutional neural networks (CNNs) can now solve some computer vision tasks at super-human levels. However, these networks only roughly mimic human visual perception. One difference from human vision is that they do not appear to perceive illusory contours (e.g. Kanizsa squares) in the same way humans do. Physiological evidence from visual cortex suggests that the perception of illusory contours could involve feedback connections. Would recurrent feedback neural networks perceive illusory contours like humans? In this work we equip a deep feedforward convolutional network with brain-inspired recurrent dynamics. The network was first pretrained with an unsupervised reconstruction objective on a natural image dataset, to expose it to natural object contour statistics. Then, a classification decision layer was added and the model was finetuned on a form discrimination task: squares vs. randomly oriented inducer shapes (no illusory contour). Finally, the model was tested with the unfamiliar "illusory contour" configuration: inducer shapes oriented to form an illusory square. Compared with feedforward baselines, the iterative "predictive coding" feedback resulted in more illusory contours being classified as physical squares. The perception of the illusory contour was measurable in the luminance profile of the image reconstructions produced by the model, demonstrating that the model really "sees" the illusion. Ablation studies revealed that natural image pretraining and feedback error correction are both critical to the perception of the illusion. Finally we validated our conclusions in a deeper network (VGG): adding the same predictive coding feedback dynamics again leads to the perception of illusory contours.

show abstract

Section: Discussionsupporting

confidence: 72%

Predictive coding feedback results in perceived illusory contours in a recurrent neural network

Pang

O’May

Choksi

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

show abstract

“…Overall, this work contributes to the general case for continuing to draw inspiration from biological visual systems in computer vision, both at the level of model architecture and dynamics. For example, in [48], we demonstrate that the same predictive coding feedback dynamics as we proposed here can help convolutional networks perceive illusory contours in a way that is more similar to humans. We believe that our user-friendly Python package Predify can open new opportunities, even for neuroscience researchers with little background in machine learning, to investigate bio-inspired hypotheses in deep computational models, and thus bridge the gap between the two communities.…”

Section: Discussionmentioning

confidence: 53%

Predify: Augmenting deep neural networks with brain-inspired predictive coding dynamics

Choksi,

Mozafari,

O'May

et al. 2021

Preprint

Self Cite

View full text Add to dashboard Cite

Deep neural networks excel at image classification, but their performance is far less robust to input perturbations than human perception. In this work we explore whether this shortcoming may be partly addressed by incorporating brain-inspired recurrent dynamics in deep convolutional networks. We take inspiration from a popular framework in neuroscience: "predictive coding". At each layer of the hierarchical model, generative feedback "predicts" (i.e., reconstructs) the pattern of activity in the previous layer. The reconstruction errors are used to iteratively update the network's representations across timesteps, and to optimize the network's feedback weights over the natural image dataset-a form of unsupervised training. We show that implementing this strategy into two popular networks, VGG16 and EfficientNetB0, improves their robustness against various corruptions. We hypothesize that other feedforward networks could similarly benefit from the proposed framework. To promote research in this direction, we provide an opensourced PyTorch-based package called Predify, which can be used to implement and investigate the impacts of the predictive coding dynamics in any convolutional neural network.

show abstract

“…Finally, artificial neural models have been shown to perceive illusory contours ( Pang et al, 2021 ; Kim et al, 2021 ), so they could also be used and even trained in the illusion meter module of the framework.…”

Section: Discussionmentioning

confidence: 99%

On the synthesis of visual illusions using deep generative models

et al. 2022

View full text Add to dashboard Cite

Visual illusions expand our understanding of the visual system by imposing constraints in the models in two different ways: i) visual illusions for humans should induce equivalent illusions in the model, and ii) illusions synthesized from the model should be compelling for human viewers too. These constraints are alternative strategies to find good vision models. Following the first research strategy, recent studies have shown that artificial neural network architectures also have human-like illusory percepts when stimulated with classical hand-crafted stimuli designed to fool humans. In this work we focus on the second (less explored) strategy: we propose a framework to synthesize new visual illusions using the optimization abilities of current automatic differentiation techniques. The proposed framework can be used with classical vision models as well as with more recent artificial neural network architectures. This framework, validated by psychophysical experiments, can be used to study the difference between a vision model and the actual human perception and to optimize the vision model to decrease this difference.

show abstract

Predictive coding feedback results in perceived illusory contours in a recurrent neural network

Cited by 38 publications

References 41 publications

Predictive coding feedback results in perceived illusory contours in a recurrent neural network

Predictive coding feedback results in perceived illusory contours in a recurrent neural network

Predify: Augmenting deep neural networks with brain-inspired predictive coding dynamics

On the synthesis of visual illusions using deep generative models

Contact Info

Product

Resources

About