Projective Simulation for Classical Learning Agents: A Comprehensive Investigation

Abstract

We study the model of projective simulation (PS), a novel approach to artificial intelligence based on stochastic processing of episodic memory which was recently introduced [1]. Here we provide a detailed analysis of the model and examine its performance, including its achievable efficiency, its learning times and the way both properties scale with the problems' dimension. In addition, we situate the PS agent in different learning scenarios, and study its learning abilities. A variety of new scenarios are being considered, thereby demonstrating the model's flexibility. Further more, to put the PS scheme in context, we compare its performance with those of Q-learning and learning classifier systems, two popular models in the field of reinforcement learning. It is shown that PS is a competitive artificial intelligence model of unique properties and strengths.

Projective simulation applied to the grid-world and the mountain-car problem

“…As shown in previous works, [1,2] the PS performance depends on the value of its internal γ and η damping parameters. In particular, it was shown that a nonzero damping parameter γ, i.e.…”

“…Following previous studies of projective simulation (PS), in which the novel model was shown to perform well on several toy-problems, [1,2] in this paper we studied the model in more challenging scenarios. In particular, we studied the performance of the PS agent in the navigation tasks of gridworld and mountain-car, in which an agent is supposed to learn how to find a goal in minimal number of steps.…”

“…[1] The recently proposed model exhibits several beneficial properties: First, it is a relatively simple model, in terms of its free parameters; [2] second, it provides a physically oriented approach toward an embodied agent design; [1] and third, the PS model, which is based on a random walk process (see below), is a natural candidate for quantization, using known methods of quantum walks, where recently an agent based on quantum projective simulation has been shown to provide a significant speed-up for certain learning scenarios. [3] From a more applied perspective, the PS model, which can be naturally applied to reinforcement learning (RL) tasks, [4][5][6] was initially tested on a number of discrete toyproblems.…”

“…[3] From a more applied perspective, the PS model, which can be naturally applied to reinforcement learning (RL) tasks, [4][5][6] was initially tested on a number of discrete toyproblems. [1,2] On such problems the PS model was shown to perform well, also in comparison with the standard models of Q-learning [4] and extended learning classifier systems. [7] In this paper, we take one step further and study the performance of the PS agent in more complicated scenarios.…”

Can Biological Quantum Networks Solve NP‐Hard Problems?

“…There is a huge number of recent papers using machine learning (ML) to train artificial neural networks to implement algorithms and protocols, for example, for classifying data. There is an excellent review article by Biamonte et al [217] which, together with a number of recent research papers on Hopfield recurrent QNNs, [218,219] quantum Boltzmann machines, [220,221] quantum circuit QNNs, [222,223] and quantum agents and artificial intelligence, [224][225][226][227][228][229] covers most of the field.…”