Adapting Behaviour via Intrinsic Reward: A Survey and Empirical Study

Abstract

Learning about many things can provide numerous benefits to a reinforcement learning system. For example, learning many auxiliary value functions, in addition to optimizing the environmental reward, appears to improve both exploration and representation learning. The question we tackle in this paper is how to sculpt the stream of experience-how to adapt the system's behaviour-to optimize the learning of a collection of value functions. A simple answer is to compute an intrinsic reward based on the statistics of each auxiliary learner, and use reinforcement learning to maximize that intrinsic reward. Unfortunately, implementing this simple idea has proven difficult, and thus has been the focus of decades of study. It remains unclear which of the many possible measures of learning would work well in a parallel learning setting where environmental reward is extremely sparse or absent. In this paper, we investigate and compare different intrinsic reward mechanisms in a new bandit-like parallel-learning testbed. We discuss the interaction between reward and prediction learners and highlight the importance of introspective prediction learners: those that increase their rate of learning when progress is possible, and decrease when it is not. We provide a comprehensive empirical comparison of 15 different rewards, including well-known ideas from reinforcement learning and active learning. Our results highlight a simple but seemingly powerful principle: intrinsic rewards based on the amount of learning can generate useful behaviour, if each individual learner is introspective.

Active World Model Learning with Progress Curiosity

“…Surprisal [Achiam and Sastry, 2017] and model disagreement [Pathak et al, 2019] present computationally-tractable alternatives to information gain, at the cost of the accuracy of the estimation. For comprehensive reviews of intrinsic motivation signal choices, see [Aubret et al, 2019, Linke et al, 2019. In this work, we present a novel method for estimating learning progress that is "consistent" with the original prediction gain objective while also scaling to high-dimensional continuous action-spaces.…”

“…Information Gain [Houthooft et al, 2016, Linke et al, 2019 based methods seek to minimize uncertainty in the Bayesian posterior distribution over model parameters:…”

“…Note that, information gain is a lower bound to the prediction gain under weak assumptions [Bellemare et al, 2016]. If the posterior has a simple form such as Laplace or Gaussian, information gain can be estimated by weight change |θ − θ| [Linke et al, 2019], and otherwise one may resort to learning a variational approximation q to approximate the information gain with D KL (q(θ )||q(θ)) [Houthooft et al, 2016]. The former weight change methods require a model after every step in the environment and is thus impractical in many settings where world model updates are expensive, e.g.…”

Review: Deep Learning in Electron Microscopy

“…Similarly, exploration can be induced by adding noise to ANN parameters 1398,1399 . Other approaches to exploration include rewarding actors for increasing action entropy [1399][1400][1401] and intrinsic motivation [1402][1403][1404] , where ANNs are incentified to explore actions that they are unsure about.…”

What Can Learned Intrinsic Rewards Capture?

“…On the other hand, many intrinsic rewards have been proposed to encourage exploration, inspired by animal behaviours. Examples include prediction error (Schmidhuber, 1991a;b;Oudeyer et al, 2007;Gordon & Ahissar, 2011;Mirolli & Baldassarre, 2013;Pathak et al, 2017), surprise (Itti & Baldi, 2006), weight change (Linke et al, 2019), and state-visitation counts (Sutton, 1990;Poupart et al, 2006;Strehl & Littman, 2008;Bellemare et al, 2016;Ostrovski et al, 2017). Although these kinds of intrinsic rewards are not domain-specific, they are often not well-aligned with the task that the agent tries to solve, and ignores the effect on the agent's learning dynamics.…”