Unsupervised Learning and Clustered Connectivity Enhance Reinforcement Learning in Spiking Neural Networks

Abstract

Reinforcement learning is a paradigm that can account for how organisms learn to adapt their behavior in complex environments with sparse rewards. To partition an environment into discrete states, implementations in spiking neuronal networks typically rely on input architectures involving place cells or receptive fields specified ad hoc by the researcher. This is problematic as a model for how an organism can learn appropriate behavioral sequences in unknown environments, as it fails to account for the unsupervised and self-organized nature of the required representations. Additionally, this approach presupposes knowledge on the part of the researcher on how the environment should be partitioned and represented and scales poorly with the size or complexity of the environment. To address these issues and gain insights into how the brain generates its own task-relevant mappings, we propose a learning architecture that combines unsupervised learning on the input projections with biologically motivated clustered connectivity within the representation layer. This combination allows input features to be mapped to clusters; thus the network self-organizes to produce clearly distinguishable activity patterns that can serve as the basis for reinforcement learning on the output projections. On the basis of the MNIST and Mountain Car tasks, we show that our proposed model performs better than either a comparable unclustered network or a clustered network with static input projections. We conclude that the combination of unsupervised learning and clustered connectivity provides a generic representational substrate suitable for further computation.

From Biological Synapses to “Intelligent” Robots

“…The work described therein makes a compelling model case for the habitual process of reinforcement learning in interaction with specific goal-directed aspects by showing that such an interaction need not be coordinated by external arbitration. The principle of self-organization [81,82] plays an important part to such effect, as clarified later with regard to unsupervised control of robot and sensor learning.…”

“…In vertebrate species, reward (reinforcement) learning consists of an agent learning specific values associated with specific states that constitute a so-called task state space [80][81][82][83][84][85][86][87][88][89]. The agent then uses the learnt knowledge to control the multiple-alternative choice of actions likely to lead to desired (reinforced) outcomes [79][80][81][82].…”

“…It has been proposed that neural networks in the mammalian orbitofrontal cortex [83][84][85][86][87][88] encode task states and task state spaces [82] during reinforcement learning. How the OFC acquires and stores this kind of information is not well understood.…”

“…How the OFC acquires and stores this kind of information is not well understood. Neural network hypotheses and models [79][80][81][82][83][84][85][86][87][88][89][90][91][92] have attempted to propose and simulate cortical candidate mechanisms inspired by known functional properties of the primate brain. Neural network models based on reservoir computing represent a suitable approach for encoding task state information during reinforcement learning [79][80][81][82].…”

“…Most reinforcement learning models account for human or animal behavior whilst assuming that the agent knows the task structure; yet, in the case of real agents (animals, humans, robots), the task structure needs to be learnt. It is critical for such a network to receive reward information as part of its input and, just as the orbitofrontal cortex receives converging sensory and reward inputs, the network is able to acquire task structure and support reinforcement learning by encoding combinations of sensory and reward events [81][82][83]. The network is based on the principles of a two-stage decision task where the agent (primate, robot) has to choose between two options A1 and A2.…”