Only Relevant Information Matters: Filtering Out Noisy Samples To Boost RL

Abstract

<jats:p>In reinforcement learning, policy gradient algorithms optimize the policy directly and rely on sampling efficiently an environment. Nevertheless, while most sampling procedures are based on direct policy sampling, self-performance measures could be used to improve such sampling prior to each policy update. Following this line of thought, we introduce SAUNA, a method where non-informative transitions are rejected from the gradient update. The level of information is estimated according to the fraction of variance explained by the value function: a measure of the discrepancy between V and the empirical returns. In this work, we use this criterion to select samples that are useful to learn from, and we demonstrate that this selection can significantly improve the performance of policy gradient methods. In this paper: (a) We introduce the SAUNA method to filter transitions. (b) We conduct experiments on a set of benchmark continuous control problems. SAUNA significantly improves performance. (c) We investigate how SAUNA reliably selects samples with the most positive impact on learning and study its improvement on both performance and sample efficiency.</jats:p>

Boosting Policy Learning in Reinforcement Learning via Adaptive Intrinsic Reward Regulation

“…Exploration, a fundamental component of reinforcement learning, significantly influences sample efficiency and agent performance. In recent years, numerous effective exploration methods have been proposed and extensively researched [7], [8], [12], [13], [15].…”

“…In response to these challenges, this paper proposes a method to adjust the parameters of intrinsic rewards based on feedback from extrinsic rewards for some existing reward schemes, such as AGAC [12], COUNT [7], and RIDE [13]. Our method tackles the exploration-exploitation trade-off by regulating intrinsic rewards, utilizing variations in the extrinsic reward during the learning process as empirical evidence to guide the adjustment of intrinsic rewards.…”

An Overview of Environmental Features that Impact Deep Reinforcement Learning in Sparse-Reward Domains

“…A landscape that, in addition, will be perfectly tailored to each procedurally generated episode of an environment. Another promising algorithm and one that deviates substantially from the former ones is AGAC (Flet-Berliac et al, 2021), which has been successfully tested in the same benchmarks as RIDE. AGAC trains a policy and a value network, as various other methods do, but critically it also trains an adversary network.…”

Learning Skills to Navigate without a Master: A Sequential Multi-Policy Reinforcement Learning Algorithm

“…The options framework [4] provides a method to automatically extract temporally extended skills for a long horizon task with the use of options, which are sub-policies that can be leveraged by some other policy in a hierarchical manner. The process of learning such temporal abstractions has been widely studied in the broad domain of hierarchical reinforcement learning [5]. In this paper, we provide an alternate approach for learning options sequentially without a higher-level policy and show a better performance on navigation tasks.…”