Reward-Free Exploration for Reinforcement Learning

Jin, Chi; Krishnamurthy, Akshay; Simchowitz, Max; Yu, Tiancheng

doi:10.48550/arxiv.2002.02794

Cited by 26 publications

(77 citation statements)

References 5 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Therefore, an important future direction is to generalize our method to reward-free settings (e.g. Jin et al, 2020).…”

Section: Conclusion Limitations and Social Impactsmentioning

confidence: 99%

See 1 more Smart Citation

Operator Deep Q-Learning: Zero-Shot Reward Transferring in Reinforcement Learning

Tang¹,

Feng²,

Liu³

2022

Preprint

View full text Add to dashboard Cite

Reinforcement learning (RL) has drawn increasing interests in recent years due to its tremendous success in various applications. However, standard RL algorithms can only be applied for single reward function, and cannot adapt to an unseen reward function quickly. In this paper, we advocate a general operator view of reinforcement learning, which enables us to directly approximate the operator that maps from reward function to value function. The benefit of learning the operator is that we can incorporate any new reward function as input and attain its corresponding value function in a zero-shot manner. To approximate this special type of operator, we design a number of novel operator neural network architectures based on its theoretical properties. Our design of operator networks outperform the existing methods and the standard design of general purpose operator network, and we demonstrate the benefit of our operator deep Q-learning framework in several tasks including reward transferring for offline policy evaluation (OPE) and reward transferring for offline policy optimization in a range of tasks.

show abstract

“…Therefore, an important future direction is to generalize our method to reward-free settings (e.g. Jin et al, 2020).…”

Section: Conclusion Limitations and Social Impactsmentioning

confidence: 99%

“…Reward Free RL Recent works on reward (task) free RL (e.g. Jin et al, 2020;Zhang et al, 2020c) break reinforcement learning into two steps: exploration phase and planning phase. In the exploration phase, they don't know the true reward functions and only focus on collecting data by exploration strategy.…”

Section: More Related Workmentioning

confidence: 99%

Operator Deep Q-Learning: Zero-Shot Reward Transferring in Reinforcement Learning

Tang¹,

Feng²,

Liu³

2022

Preprint

View full text Add to dashboard Cite

show abstract

“…For usual MDPs, states that cannot be reached pose no problems: we do not need to learn transition or reward probabilities of these states, since no optimal policy will use that state. Indeed, with this observation at hand, reward free exploration techniques [31,33] can be utilized to learn a good model w.r.t. all possible reward functions (given a fixed initial distribution).…”

Section: Notationmentioning

confidence: 99%

“…In contrast to fully observable environments, very little is known about the exploration in partially observable MDPs (POMDPs). In general, RL in POMDPs may require an exponential number of samples (without simplifying structural assumptions) [35,31]. Therefore, it is important to consider natural sub-classes of POMDPs which admit tractable solutions.…”

Section: Introductionmentioning

confidence: 99%

Reinforcement Learning in Reward-Mixing MDPs

Kwon¹,

Efroni²,

Caramanis³

et al. 2021

Preprint

View full text Add to dashboard Cite

Learning a near optimal policy in a partially observable system remains an elusive challenge in contemporary reinforcement learning. In this work, we consider episodic reinforcement learning in a reward-mixing Markov decision process (MDP). There, a reward function is drawn from one of multiple possible reward models at the beginning of every episode, but the identity of the chosen reward model is not revealed to the agent. Hence, the latent state space, for which the dynamics are Markovian, is not given to the agent. We study the problem of learning a near optimal policy for two reward-mixing MDPs. Unlike existing approaches that rely on strong assumptions on the dynamics, we make no assumptions and study the problem in full generality. Indeed, with no further assumptions, even for two switching reward-models, the problem requires several new ideas beyond existing algorithmic and analysis techniques for efficient exploration. We provide the first polynomial-time algorithm that finds an ǫ-optimal policy after exploring Õ(poly(H, ǫ −1 ) • S 2 A 2 ) episodes, where H is time-horizon and S, A are the number of states and actions respectively. This is the first efficient algorithm that does not require any assumptions in partially observed environments where the observation space is smaller than the latent state space.Algorithm 1 Learning Two Reward-Mixture MDPs 1: Run pure-exploration (Algorithm 2) to estimate second-order correlations 2: Estimate 2RM-MDP parameters M from the collected data (Algorithm 3) 3: Return π, the (approximately) optimal policy of M P m the probability of any event measured in the m th context (or in m th MDP). If probability of an event depends on a policy π, we add superscript π to P. We denote V π M as an expected long-term reward for model M with policy π. We use • to denote empirical counterparts. For any set A and d ∈ N, A d is a d-ary Cartesian product over A. We use a ∨ b to refer max(a, b) and a ∧ b to mean min(a, b) for a, b ∈ R. Specifically for M = 2: We use shorthand p m (x) := R m (r = 1|x) for m = 1, 2. Let an expected averaged reward p + (x) := 1 2 (p 1 (x) + p 2 (x)), differences in rewards p − (x) := 1 2 (p 1 (x) − p 2 (x)) and ∆(x) := |p − (x)|. AlgorithmsBefore developing a learning algorithm for 2RM-MDP, let us provide intuition behind our algorithm. At a high-level, our approach lies on the following observation:The latent reward model of 2RM-MDP can be recovered from reward correlations and the averaged reward.

show abstract

“…Unsupervised exploration is an emergent and challenging topic for reinforcement learning (RL) that inspires research interests in both application [Riedmiller et al, 2018, Finn and Levine, 2017, Xie et al, 2018, Schaul et al, 2015, Riedmiller et al, 2018 and theory [Hazan et al, 2018, Jin et al, 2020, Zhang et al, 2020a, Zhang et al, 2020b, Wu et al, 2020, Wang et al, 2020b. The formal formulation of an unsupervised RL problem consists of an exploration phase and a planning phase [Jin et al, 2020]: in the exploration phase, an agent interacts with the unknown environment without the supervision of reward signals; then in the planning phase, the agent is prohibited to interact with the environment, and is required to compute a nearly optimal policy for some revealed reward function based on its exploration experiences. In particular, if the reward function is fixed yet unknown during exploration, the problem is called task-agnostic exploration (TAE) [Zhang et al, 2020a], and if the reward function is allowed to be chosen arbitrary, the problem is called reward-free exploration (RFE) [Jin et al, 2020].…”

Section: Introductionmentioning

confidence: 99%

Gap-Dependent Unsupervised Exploration for Reinforcement Learning

Wu¹,

Braverman²,

Yang³

2021

Preprint

View full text Add to dashboard Cite

For the problem of task-agnostic reinforcement learning (RL), an agent first collects samples from an unknown environment without the supervision of reward signals, then is revealed with a reward and is asked to compute a corresponding nearoptimal policy. Existing approaches mainly concern the worst-case scenarios, in which no structural information of the reward/transition-dynamics is utilized. Therefore the best sample upper bound is ∝ O(1/ 2 ), where > 0 is the target accuracy of the obtained policy, and can be overly pessimistic. To tackle this issue, we provide an efficient algorithm that utilizes a gap parameter, ρ > 0, to reduce the amount of exploration. In particular, for an unknown finite-horizon Markov decision process, the algorithm takes only O(1/ • (H 3 SA/ρ + H 4 S 2 A)) episodes of exploration, and is able to obtain an -optimal policy for a post-revealed reward with sub-optimality gap at least ρ, where S is the number of states, A is the number of actions, and H is the length of the horizon, obtaining a nearly quadratic saving in terms of . We show that, information-theoretically, this bound is nearly tight for ρ < Θ(1/(HS)) and H > 1. We further show that ∝ O(1) sample bound is possible for H = 1 (i.e., multi-armed bandit) or with a sampling simulator, establishing a stark separation between those settings and the RL setting.Preprint. Under review.

show abstract

Reward-Free Exploration for Reinforcement Learning

Cited by 26 publications

References 5 publications

Operator Deep Q-Learning: Zero-Shot Reward Transferring in Reinforcement Learning

Operator Deep Q-Learning: Zero-Shot Reward Transferring in Reinforcement Learning

Reinforcement Learning in Reward-Mixing MDPs

Gap-Dependent Unsupervised Exploration for Reinforcement Learning

Contact Info

Product

Resources

About