Adaptive Reward-Free Exploration

Kaufmann, Emilie; Ménard, Pierre; Domingues, Omar Darwiche; Jönsson, Anders; Leurent, Edouard; Valko, Michal

doi:10.48550/arxiv.2006.06294

Cited by 4 publications

(12 citation statements)

References 11 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Note this framework captures many practical MORL applications including the previous safe autonomous driving applications. It also extends the recent proposed reward-free exploration in reinforcement learning [Jin et al, 2020, Zhang et al, 2020a, Kaufmann et al, 2020, Wang et al, 2020 to MORL cases.…”

Section: Introductionmentioning

confidence: 70%

See 1 more Smart Citation

Accommodating Picky Customers: Regret Bound and Exploration Complexity for Multi-Objective Reinforcement Learning

Wu¹,

Braverman²,

Yang³

2020

Preprint

View full text Add to dashboard Cite

In this paper we consider multi-objective reinforcement learning where the objectives are balanced using preferences. In practice, the preferences are often given in an adversarial manner, e.g., customers can be picky in many applications. We formalize this problem as an episodic learning problem on a Markov decision process, where transitions are unknown and a reward function is the inner product of a preference vector with pre-specified multi-objective reward functions. In the online setting, the agent receives a (adversarial) preference every episode and proposes policies to interact with the environment. We provide a model-based algorithm that achieves a regret bound O min{d, S} • H 3 SAK , where d is the number of objectives, S is the number of states, A is the number of actions, H is the length of the horizon, and K is the number of episodes. Furthermore, we consider preference-free exploration, i.e., the agent first interacts with the environment without specifying any preference and then is able to accommodate arbitrary preference vectors up to ǫ error. Our proposed algorithm is provably efficient with a nearly optimal sample complexity O min{d,S}•H 4 SA ǫ 2 .

show abstract

Section: Introductionmentioning

confidence: 70%

“…In this section we study MORL in the preference-free exploration (PFE) setting. We note that PFE is the counter part of reward-free exploration (RFE) [Jin et al, 2020, Kaufmann et al, 2020, Zhang et al, 2020a, Wang et al, 2020 in MORL. PFE consists two phases.…”

Section: Preference-free Explorationmentioning

confidence: 99%

Accommodating Picky Customers: Regret Bound and Exploration Complexity for Multi-Objective Reinforcement Learning

Wu¹,

Braverman²,

Yang³

2020

Preprint

View full text Add to dashboard Cite

show abstract

“…It requires high probability guarantee for learning optimal policy for any reward function, which is strictly stronger than the standard learning task that one only needs to learn to optimal policy for a fixed reward. Later, Kaufmann et al (2020); Menard et al (2020) establish the Õ(H 3 S 2 A/ǫ 2 ) complexity and Zhang et al (2020d) further tightens the dependence to Õ(H 2 S 2 A/ǫ 2 ). 12 Recently, Zhang et al (2020c) proposes the task-agnostic setting where one needs to use exploration data to simultaneously learn K tasks and provides a upper bound with complexity Õ(H 5 SA log(K)/ǫ 2 ).…”

Section: Discussionmentioning

confidence: 91%

Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settings

Yin,

Wang

2021

Preprint

View full text Add to dashboard Cite

This work studies the statistical limits of uniform convergence for offline policy evaluation (OPE) problems with model-based methods (for finite horizon MDP) and provides a unified view towards optimal learning for several well-motivated offline tasks. Uniform OPE sup Π |Q π − Qπ | < ǫ (initiated by Yin et al. (2021b)) is a stronger measure than the point-wise (fixed policy) OPE and ensures offline policy learning when Π contains all policies (global policy class). In this paper, we establish an Ω(H 2 S/d m ǫ 2 ) lower bound (over model-based family) for the global uniform OPE, where d m is the minimal state-action probability induced by the behavior policy. Next, our main result establishes an episode complexity of Õ(H 2 /d m ǫ 2 ) for local uniform convergence that applies to all near-empirically optimal policies for the MDPs with stationary transition. This result implies the optimal sample complexity for offline learning and separates the local uniform OPE from the global case due to the extra S factor. Paramountly, the model-based method combining with our new analysis technique (singleton absorbing MDP) can be adapted to the new settings: offline task-agnostic and the offline reward-free with optimal complexity Õ(H 2 log(K)/d m ǫ 2 ) (K is the number of tasks) and Õ(H 2 S/d m ǫ 2 ) respectively, which provides a unified framework for simultaneously solving different offline RL problems.

show abstract

“…For the single-agent scenario, Jin et al (2020a) formalizes the reward-free RL for the tabular setting and provide theoretical analysis for the proposed algorithm with an O(poly(H, |S|, |A|)/ε 2 ) sample complexity for achieving ε-suboptimal policy. The sample complexity for the tabular setting is further improved in several recent works (Kaufmann et al, 2020;Ménard et al, 2020;Zhang et al, 2020). Recently, Zanette et al (2020b); Wang et al (2020a) study the reward-free RL from the perspective of the linear function approximation.…”

Section: Related Workmentioning

confidence: 99%

“…Recently, many works focus on designing provably sample-efficient reward-free RL algorithms. For the single-agent tabular case, Jin et al (2020a); Kaufmann et al (2020); Ménard et al (2020); Zhang et al (2020) achieve O(poly(H, |S|, |A|)/ε 2 ) sample complexity for obtaining ε-suboptimal policy, where |S|, |A| are the sizes of state and action space, respectively. In view of the large action and state spaces, the works Zanette et al (2020b); Wang et al (2020a) theoretically analyze reward-free RL by applying the linear function approximation for the single-agent Markov decision process (MDP), which achieve O(poly(H, d)/ε 2 ) sample complexity with d denoting the dimension of the feature space.…”

Section: Introductionmentioning

confidence: 99%

On Reward-Free RL with Kernel and Neural Function Approximations: Single-Agent MDP and Markov Game

Qiu

Wang³

et al. 2021

Preprint

View full text Add to dashboard Cite

To achieve sample efficiency in reinforcement learning (RL), it necessitates efficiently exploring the underlying environment. Under the offline setting, addressing the exploration challenge lies in collecting an offline dataset with sufficient coverage. Motivated by such a challenge, we study the reward-free RL problem, where an agent aims to thoroughly explore the environment without any pre-specified reward function. Then, given any extrinsic reward, the agent computes the policy via a planning algorithm with offline data collected in the exploration phase. Moreover, we tackle this problem under the context of function approximation, leveraging powerful function approximators. Specifically, we propose to explore via an optimistic variant of the value-iteration algorithm incorporating kernel and neural function approximations, where we adopt the associated exploration bonus as the exploration reward. Moreover, we design exploration and planning algorithms for both single-agent MDPs and zero-sum Markov games and prove that our methods can achieve O(1/ε 2 ) sample complexity for generating a ε-suboptimal policy or ε-approximate Nash equilibrium when given an arbitrary extrinsic reward. To the best of our knowledge, we establish the first provably efficient reward-free RL algorithm with kernel and neural function approximators.

show abstract

Adaptive Reward-Free Exploration

Cited by 4 publications

References 11 publications

Accommodating Picky Customers: Regret Bound and Exploration Complexity for Multi-Objective Reinforcement Learning

Accommodating Picky Customers: Regret Bound and Exploration Complexity for Multi-Objective Reinforcement Learning

Optimal Uniform OPE and Model-based Offline Reinforcement Learning in Time-Homogeneous, Reward-Free and Task-Agnostic Settings

On Reward-Free RL with Kernel and Neural Function Approximations: Single-Agent MDP and Markov Game

Contact Info

Product

Resources

About