Self-Supervised Online Reward Shaping in Sparse-Reward Environments

Abstract

We propose a novel reinforcement learning framework that performs self-supervised online reward shaping, yielding faster, sample efficient performance in sparse reward environments. The proposed framework alternates between updating a policy and inferring a reward function. While the policy update is done with the inferred, potentially dense reward function, the original sparse reward is used to provide a self-supervisory signal for the reward update by serving as an ordering over the observed trajectories. The proposed framework is based on the theory that altering the reward function does not affect the optimal policy of the original MDP as long as we maintain certain relations between the altered and the original reward. We name the proposed framework ClAssification-based REward Shaping (CaReS), since we learn the altered reward in a self-supervised manner using classifier based reward inference. Experimental results on several sparse-reward environments demonstrate that the proposed algorithm is not only significantly more sample efficient than the state-of-the-art baseline, but also achieves a similar sample efficiency to MDPs that use hand-designed dense reward functions.

Robust Generative Adversarial Imitation Learning via Local Lipschitzness

“…Number of PPO epochs: 10. Number of projected gradient ascent steps to compute δ s and δ s,a through ( 9), (12) in the main text: 10 steps. PPO clipping parameter: 0.2.…”

“…We perform two sets of experiments, one set uses the L 2 norm and the other uses L ∞ norm throughout the experiments. The norms are used for the following: 1) Defining the balls in which we find the adversarial perturbations δ s and δ s,a through ( 9), (12) in the main text. 2) Defining the ball from which we sample the noise injected at test time.…”

“…Note that δ s is the perturbation within the L 2 ball of radius r p which causes the largest divergence in the policy. Since it is computationally infeasible to find the exact solution to (12) for every sate, we instead use projected gradient ascent steps to get close to the solution. GAIL uses TRPO [15] steps to update the generator.…”

“…For a policy π, the discounted causal entropy is defined as H(π) := E (s,a)∈ρπ [− log(π(a|s)/(1 − γ)] in which ρ π is the state-action distribution induced by policy π. Generative Adversarial Imitation Learning. Imitation learning algorithms [6,7,8,9,10,11,12] aim to learn a policy that mimics the underlying behavior of the demonstrations. Methods such as inverse reinforcement learning (IRL) [7,8,9] do so by learning a reward function as an intermediate step.…”

A reinforcement learning path planning approach for range-only underwater target localization with autonomous vehicles

“…However, such approach can easily exploit badly designed rewards, and get stuck in local optima and induce behavior that the designer did not intend. In contrast, goal-based sparse rewards are appealing since they do not suffer from the reward exploration problem [32]. In addition, this simple small set of rules have its similarities with biological behaviours, and therefore, applicable to animals with very limited level of information processing [33].…”