Predify: Augmenting deep neural networks with brain-inspired predictive coding dynamics

Abstract

Deep neural networks excel at image classification, but their performance is far less robust to input perturbations than human perception. In this work we explore whether this shortcoming may be partly addressed by incorporating brain-inspired recurrent dynamics in deep convolutional networks. We take inspiration from a popular framework in neuroscience: "predictive coding". At each layer of the hierarchical model, generative feedback "predicts" (i.e., reconstructs) the pattern of activity in the previous layer. The reconstruction errors are used to iteratively update the network's representations across timesteps, and to optimize the network's feedback weights over the natural image dataset-a form of unsupervised training. We show that implementing this strategy into two popular networks, VGG16 and EfficientNetB0, improves their robustness against various corruptions. We hypothesize that other feedforward networks could similarly benefit from the proposed framework. To promote research in this direction, we provide an opensourced PyTorch-based package called Predify, which can be used to implement and investigate the impacts of the predictive coding dynamics in any convolutional neural network.

Bio-inspired neural networks implement different recurrent visual processing strategies than task-trained ones do

“…At run time, this reconstructed activity is then added to the true activity at the first layer. While this form of predictive processing differs from the more common implementation of predictive coding (Rao & Ballard, 1999) in that it does not propagate error signals, it has been shown to increase performance on classification of images with pixel noise (Choksi et al, 2020(Choksi et al, , 2021. For the lateral connections, we implement within-feature spatial surround suppression and nearby facilitation by applying a one channel convolutional filter defined by a difference of Gaussians (Figure 1F), inspired by Hasani et al (2019) (though note in their study the full model is trained with this filter present, which is not the case here).…”

“…We try out different ways of adding both feedback and lateral connections. First, we use bio-inspired methods: we add lateral connections that implement surround suppression, which is found throughout the visual system (Angelucci et al, 2017;Hasani, Soleymani, & Aghajan, 2019); for feedback connections, we implement a modified predictive processing scheme (Choksi et al, 2020(Choksi et al, , 2021Mumford, 1992;Rao & Ballard, 1999). To add connections inspired mainly by the function we know recurrence to play, we also add lateral and feedback connections trained with backpropagation to classify noisy images.…”

Energy Optimization Induces Predictive-coding Properties in a Multicompartment Spiking Neural Network Model

“…these specifically hardwired microcircuits are precisely present in the cortex. The second major limitation is that most models are non-spiking networks which lack biological realism [14,15,16,10]. This has been mostly due to the lack of a straightforward way to transfer classical rate-based predictive coding to a spiking implementation, ie.…”

“…The difficulties in training spiking neural networks have also hindered efforts in this direction [5]. Additional trade-offs occur between biological fidelity and scalability, which makes it difficult to study more complex phenomena in a biological network [10,12,16,15]. The few works implementing predictive coding in spiking neural networks like in [12] leave a gap in the literature for more biologically realistic network models without specific architectural biases.…”

Disrupted visual input unveils the computational details of artificial neural networks for face perception

“…Some recent studies indicated that a renewed focus on the inspiration of the human nervous system for neural networks could be more helpful for the further improvement and development of artificial intelligence. For example, the deep network PredNet (Lotter et al, 2016 ) and the neural network robustness enhancement package Predify (Choksi et al, 2021 ) are inspired by the concept of predictive coding observed in neuroscience. In addition, the brain-inspired replay method effectively solves the problem of catastrophic forgetting of artificial neural networks (van de Ven et al, 2020 ).…”