Surprise-Based Intrinsic Motivation for Deep Reinforcement Learning

Abstract

Exploration in complex domains is a key challenge in reinforcement learning, especially for tasks with very sparse rewards. Recent successes in deep reinforcement learning have been achieved mostly using simple heuristic exploration strategies such as -greedy action selection or Gaussian control noise, but there are many tasks where these methods are insufficient to make any learning progress. Here, we consider more complex heuristics: efficient and scalable exploration strategies that maximize a notion of an agent's surprise about its experiences via intrinsic motivation. We propose to learn a model of the MDP transition probabilities concurrently with the policy, and to form intrinsic rewards that approximate the KL-divergence of the true transition probabilities from the learned model. One of our approximations results in using surprisal as intrinsic motivation, while the other gives the k-step learning progress. We show that our incentives enable agents to succeed in a wide range of environments with high-dimensional state spaces and very sparse rewards, including continuous control tasks and games in the Atari RAM domain, outperforming several other heuristic exploration techniques.

Is Curiosity All You Need? On the Utility of Emergent Behaviours from Curious Exploration

“…The utilisation of forward modelling error as reward signal has been implemented in deterministic and probabilistic settings (Achiam and Sastry, 2017;Shelhamer et al, 2016) and is commonly known as curiosity learning. The error reward signal encourages the exploration of unfamiliar parts of the state-action space which are not yet wellpredictable.…”

Towards Robust Bisimulation Metric Learning

“…This is often done by providing intrinsic motivation via a self-derived reward, resulting in "curiosity-driven" behaviour [8,4]. Such approaches include surprise [1,5] (where experiencing unexpected dynamics is rewarded), and empowerment [40,20] (where the agent prefers states in which it has more control).…”

“…On the other hand, we can show that the formulation of the metric learning loss is susceptible to embedding explosion if the representation space is left unconstrained 2 . In our work, we build upon the DBC model in an attempt to tackle both problems: (1) we address embedding explosion by stabilizing the state representation space via a norm constraint and (2) we prevent embedding collapse by altering the encoder training method.…”

A Max-Min Entropy Framework for Reinforcement Learning

“…Exploration in RL: Exploration is one of the most important issues in model-free RL, as there is the key assumption that all state-action pairs must be visited infinitely often to guarantee the convergence of Q-function [56]. In order to explore diverse state-action pairs in the joint state-action space, various methods have been considered in prior works: intrinsically-motivated reward based on curiosity [5,11], model prediction error [1,10], information gain [26,28,29], and counting states [33,35]. These exploration techniques improve exploration and performance in challenging sparse-reward environments [3,10,13].…”

“…Starting from the left-lower corner (0.5, 0.5), the agent explores the maze without any external reward. First, note that for this pure exploration task, the optimal policy maximizing J M axEnt (π) is given by the uniform policy that selects all actions in 1] uniformly regardless of the value of s t . This is because the uniform distribution has maximum entropy for a bounded space [14].…”