Iterative free-energy optimization for recurrent neural networks (INFERNO)

Abstract

The intra-parietal lobe coupled with the Basal Ganglia forms a working memory that demonstrates strong planning capabilities for generating robust yet flexible neuronal sequences. Neurocomputational models however, often fails to control long range neural synchrony in recurrent spiking networks due to spontaneous activity. As a novel framework based on the free-energy principle, we propose to see the problem of spikes’ synchrony as an optimization problem of the neurons sub-threshold activity for the generation of long neuronal chains. Using a stochastic gradient descent, a reinforcement signal (presumably dopaminergic) evaluates the quality of one input vector to move the recurrent neural network to a desired activity; depending on the error made, this input vector is strengthened to hill-climb the gradient or elicited to search for another solution. This vector can be learned then by one associative memory as a model of the basal-ganglia to control the recurrent neural network. Experiments on habit learning and on sequence retrieving demonstrate the capabilities of the dual system to generate very long and precise spatio-temporal sequences, above two hundred iterations. Its features are applied then to the sequential planning of arm movements. In line with neurobiological theories, we discuss its relevance for modeling the cortico-basal working memory to initiate flexible goal-directed neuronal chains of causation and its relation to novel architectures such as Deep Networks, Neural Turing Machines and the Free-Energy Principle.

Gated spiking neural network using Iterative Free-Energy Optimization and rank-order coding for structure learning in memory sequences (INFERNO GATE)

“…We showed in [1] that this variational process is similar to a stochastic descent gradient algorithm performed iteratively. We here add a more sophisticated gradient descent algorithm corresponding to a simulated annealing mechanism in order to account for the neuromodulators involved in decision-making in the PFC for uncertainty and surprise [16,95].…”

“…In this paper, we propose to use the neural architecture Inferno, standing for Iterative Free-Energy Optimization in Recurrent Neural Network, for the learning of temporal patterns and the serial recall of sequences [1,2]. We originally proposed this neuronal architecture to model the cortico-basal ganglia loop [1] for retrieving motor and audio primitives using Spike Timing-dependent Plasticity (STDP) within the framework of predictive coding and free-energy minimization [3,4,5]. Here, we propose to implement a similar free-energy minimization network but this time in the prefrontal-basal ganglia loop for the serial recall of memory sequences and for the learning of temporal pattern primitives, using gain-modulation instead of STDP.…”

“…The neural architecture Inferno Gate reproduces the main configuration of the original Inferno network [1] with two coupled learning systems that minimize their mutual prediction error (the free-energy), see In order to minimize error, the second network (dlPFC/OFC) generates intrinsic noise I noise to control the dynamics of the first one (lPFC/dlPFC) following a reinforcement learning (RL) mechanism. In Fig.…”

“…The rank-order coding (ROC) algorithm has been proposed by Thorpe and colleagues to model the information processing performed in the Visual Cortex by feedforward integrate-and-fire neurons [96]. We have expanded their use to recurrent neural network models in [97,1] replicating the Spike Timing-Dependent Plasticity learning mechanism.…”

INFERNO: A Novel Architecture for Generating Long Neuronal Sequences with Spikes

“…The minimization of the free energy means to predict for one particular policy its expected state and to optimize it over time in order to minimize future errors [4]. Our neural model is based on this principle of Iterative Free-Energy Optimization for Recurrent Neural Networks, and we named it INFERNO [9], see Fig. 1.…”