Offline Reinforcement Learning as Anti-Exploration

Rezaeifar, Shideh; Dadashi, Robert; Vieillard, Nino; Hussenot, Léonard; Bachem, Olivier; Pietquin, Olivier; Geist, Matthieu

doi:10.48550/arxiv.2106.06431

Cited by 4 publications

(4 citation statements)

References 25 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Recently, there have been many offline RL methods proposed to implement such a penalty, by introducing support constrains [Fujimoto, Meger, and Precup 2019, Ghasemipour, Schuurmans, and Gu 2021 or policy regularization [Kostrikov, Nair, and Levine 2021, Fujimoto and Gu 2021, Peng et al 2019, Wu, Tucker, and Nachum 2019, Kumar et al 2019a. Alternatively, some offline RL methods introduce the uncertainty estimation [Rezaeifar et al 2021, Wu et al 2021, Ma, Jayaraman, and Bastani 2021, Bai et al 2022 or the conservation [Kumar et al 2020, Lyu et al 2022] over values to overcome the potential overestimation for OOD state actions. In the same spirit, model-based offline RL methods similarly employ the distribution (correcting) regularization [Hishinuma and Senda 2021, uncertain estimation [Kidambi et al 2020, and value conservation [Yu et al 2021] to eliminate the OOD issues.…”

Section: Related Workmentioning

confidence: 99%

Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning

Liu,

Zhang,

Wei

et al. 2024

AAAI

View full text Add to dashboard Cite

Offline reinforcement learning (RL) aims to learn a policy using only pre-collected and fixed data. Although avoiding the time-consuming online interactions in RL, it poses challenges for out-of-distribution (OOD) state actions and often suffers from data inefficiency for training. Despite many efforts being devoted to addressing OOD state actions, the latter (data inefficiency) receives little attention in offline RL. To address this, this paper proposes the cross-domain offline RL, which assumes offline data incorporate additional source-domain data from varying transition dynamics (environments), and expects it to contribute to the offline data efficiency. To do so, we identify a new challenge of OOD transition dynamics, beyond the common OOD state actions issue, when utilizing cross-domain offline data. Then, we propose our method BOSA, which employs two support-constrained objectives to address the above OOD issues. Through extensive experiments in the cross-domain offline RL setting, we demonstrate BOSA can greatly improve offline data efficiency: using only 10% of the target data, BOSA could achieve 74.4% of the SOTA offline RL performance that uses 100% of the target data. Additionally, we also show BOSA can be effortlessly plugged into model-based offline RL and noising data augmentation techniques (used for generating source-domain data), which naturally avoids the potential dynamics mismatch between target-domain data and newly generated source-domain data.

show abstract

Section: Related Workmentioning

confidence: 99%

Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning

Liu,

Zhang,

Wei

et al. 2024

AAAI

View full text Add to dashboard Cite

show abstract

“…Previous model-free offline RL algorithms typically rely on policy constraints to restrict the learned policy from producing the OOD actions. In particular, previous works add behavior cloning (BC) loss in policy training (Fujimoto et al, 2019;Fujimoto & Gu, 2021;Ghasemipour et al, 2021), measure the divergence between the behavior policy and the learned policy (Kumar et al, 2019;Kostrikov et al, 2021), apply advantage-weighted constraints to balance BC and advantages (Siegel et al, 2020;Wang et al, 2020b), penalize the prediction-error of a variational auto-encoder (Rezaeifar et al, 2021), and learn latent actions (or primitives) from the offline data (Zhou et al, 2020;Ajay et al, 2021). Nevertheless, such methods may cause overly conservative value functions and are easily affected by the behavior policy (Nair et al, 2020;Lee et al, 2021b).…”

Section: Related Workmentioning

confidence: 99%

Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning

Bai¹,

Wang²,

Yang³

et al. 2022

Preprint

View full text Add to dashboard Cite

Offline Reinforcement Learning (RL) aims to learn policies from previously collected datasets without exploring the environment. Directly applying off-policy algorithms to offline RL usually fails due to the extrapolation error caused by the out-of-distribution (OOD) actions. Previous methods tackle such problems by penalizing the Q-values of OOD actions or constraining the trained policy to be close to the behavior policy. Nevertheless, such methods typically prevent the generalization of value functions beyond the offline data and also lack a precise characterization of OOD data. In this paper, we propose Pessimistic Bootstrapping for offline RL (PBRL), a purely uncertainty-driven offline algorithm without explicit policy constraints. Specifically, PBRL conducts uncertainty quantification via the disagreement of bootstrapped Q-functions, and performs pessimistic updates by penalizing the value function based on the estimated uncertainty. To tackle the extrapolating error, we further propose a novel OOD sampling method. We show that such OOD sampling and pessimistic bootstrapping yields a provable uncertainty quantifier in linear MDPs, thus providing the theoretical underpinning for PBRL. Extensive experiments on D4RL benchmark show that PBRL has better performance compared to the state-of-the-art algorithms.

show abstract

“…Our work follows the latter line (also known as the principle of pessimism), which has garnered significant attention recently. In fact, pessimism has been incorporated into recent development of various offline RL approaches, such as policy-based approaches (Rezaeifar et al, 2021;Xie et al, 2021a;Zanette et al, 2021), model-based approaches (Jin et al, 2021;Kidambi et al, 2020;Rashidinejad et al, 2021;Xie et al, 2021b;Yin and Wang, 2021;Yin et al, 2022;Yu et al, 2021bYu et al, , 2020, and model-free approaches (Kumar et al, 2020;Yan et al, 2022;Yu et al, 2021a).…”

Section: Related Workmentioning

confidence: 99%

Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity

Shi¹,

Li²,

Wei³

et al. 2022

Preprint

View full text Add to dashboard Cite

Offline or batch reinforcement learning seeks to learn a near-optimal policy using history data without active exploration of the environment. To counter the insufficient coverage and sample scarcity of many offline datasets, the principle of pessimism has been recently introduced to mitigate high bias of the estimated values. While pessimistic variants of model-based algorithms (e.g., value iteration with lower confidence bounds) have been theoretically investigated, their model-free counterparts -which do not require explicit model estimation -have not been adequately studied, especially in terms of sample efficiency. To address this inadequacy, we study a pessimistic variant of Q-learning in the context of finite-horizon Markov decision processes, and characterize its sample complexity under the singlepolicy concentrability assumption which does not require the full coverage of the state-action space. In addition, a variance-reduced pessimistic Q-learning algorithm is proposed to achieve near-optimal sample complexity. Altogether, this work highlights the efficiency of model-free algorithms in offline RL when used in conjunction with pessimism and variance reduction.

show abstract

Offline Reinforcement Learning as Anti-Exploration

Cited by 4 publications

References 25 publications

Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning

Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning

Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning

Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexity

Contact Info

Product

Resources

About