Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning

Bai, Chenjia; Wang, Lingxiao; Yang, Zhuoran; Deng, Zhihong; Garg, Animesh; Liu, Peng; Wang, Zhaoran

doi:10.48550/arxiv.2202.11566

Cited by 9 publications

(19 citation statements)

References 6 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…Pessimism in batch RL serves as the main tool for conservative policy optimization by quantifying the uncertainty of the estimation and discouraging the exploration of the learned policy from visiting the less explored state-action pair in the batch data. The success of the pessimsitic-typed algorithms has been demonstrated in many applications (e.g., Kumar et al, 2019;Bai et al, 2022). In terms of the data coverage, instead of the full-coverage assumption (i.e., dπ b T is uniformly bounded away from 0) required by many classic RL algorithms, algorithms with a proper degree of pessimism only require that the in-class optimal policy π * is covered by the behavior one, which is thus more desirable.…”

Section: Policy Learning In Batch Rlmentioning

confidence: 99%

STEEL: Singularity-aware Reinforcement Learning

Chen¹,

Qi²,

Wan³

2023

Preprint

View full text Add to dashboard Cite

Batch reinforcement learning (RL) aims at finding an optimal policy in a dynamic environment in order to maximize the expected total rewards by leveraging pre-collected data. A fundamental challenge behind this task is the distributional mismatch between the batch data generating process and the distribution induced by target policies. Nearly all existing algorithms rely on the absolutely continuous assumption on the distribution induced by target policies with respect to the data distribution, so that the batch data can be used to calibrate target policies via the change of measure. However, the absolute continuity assumption could be violated in practice, especially when the state-action space is large or continuous. In this paper, we propose a new batch RL algorithm without requiring absolute continuity in the setting of an infinite-horizon Markov decision process with continuous states and actions. We call our algorithm STEEL: SingulariTy-awarE rEinforcement Learning. Our algorithm is motivated by a new error analysis on off-policy evaluation, where we use maximum mean discrepancy, together with distributionally robust optimization, to characterize the error of off-policy evaluation caused by the possible singularity and to enable the power of model extrapolation. By leveraging the idea of pessimism and under some mild conditions, we derive a finite-sample regret guarantee for our proposed algorithm without imposing absolute continuity. Compared with existing algorithms, STEEL only requires some minimal data-coverage assumption and thus greatly enhances the applicability and robustness of batch RL. Extensive simulation studies and one real experiment on personalized pricing demonstrate the superior performance of our method when facing possible singularity in batch RL.

show abstract

Section: Policy Learning In Batch Rlmentioning

confidence: 99%

STEEL: Singularity-aware Reinforcement Learning

Chen¹,

Qi²,

Wan³

2023

Preprint

View full text Add to dashboard Cite

show abstract

“…There is a growing number of results under partial coverage following the principle of pessimism in offline RL (Yu et al, 2020;Kidambi et al, 2020). In comparison to works that focus on tabular (Rashidinejad et al, 2021;Shi et al, 2022;Yin and Wang, 2021) or linear models (Jin et al, 2020;Chang et al, 2021;Zhang et al, 2022;Nguyen-Tang et al, 2022;Bai et al, 2022), our emphasis is on general function approximation (Jiang and Huang, 2020;Uehara and Sun, 2022;Xie et al, 2021;Zhan et al, 2022;Rashidinejad et al, 2022;Zanette and Wainwright, 2022). Among them, we specifically focus on model-free methods.…”

Section: Related Workmentioning

confidence: 99%

Refined Value-Based Offline RL under Realizability and Partial Coverage

Uehara¹,

Kallus²,

Lee³

et al. 2023

Preprint

View full text Add to dashboard Cite

In offline reinforcement learning (RL) we have no opportunity to explore so we must make assumptions that the data is sufficient to guide picking a good policy, taking the form of assuming some coverage, realizability, Bellman completeness, and/or hard margin (gap). In this work we propose value-based algorithms for offline RL with PAC guarantees under just partial coverage, specifically, coverage of just a single comparator policy, and realizability of soft (entropy-regularized) Q-function of the single policy and a related function defined as a saddle point of certain minimax optimization problem. This offers refined and generally more lax conditions for offline RL. We further show an analogous result for vanilla Q-functions under a soft margin condition. To attain these guarantees, we leverage novel minimax learning algorithms to accurately estimate soft or vanilla Q-functions with L 2 -convergence guarantees. Our algorithms' loss functions arise from casting the estimation problems as nonlinear convex optimization problems and Lagrangifying.

show abstract

“…To summarize, the algorithm alternates between collecting on-policy data samples in AMG and updating the function approximators. In detail, the latter procedure includes updating the Q-value with (11), updating the optimized policy with (14), and updating the target Q-value as well as the reference policy with the moving-average rule. The complete algorithm is listed in Appendix D.…”

Section: Offline Reinforcement Learning With Pessimism-modulated Dyna...mentioning

confidence: 99%

“…Learning within the data manifold limits the degree to which the policy improves, and recent works attempt to relieve the restriction. Along the model-free line, EDAC [13] and PBRL [14] quantify uncertainty of Q-value via neural network ensemble, and assign penalty to Q-value depending on the uncertainty degree. In this way, the OOD state-action pairs are touchable if they pose low uncertainty on Q-value.…”

Section: Related Workmentioning

confidence: 99%

“…While they achieve remarkable performance gains, the policy regularizer and the Q-value penalty tightly restricts the produced policy within the data manifold. Instead, more recent works consider to quantify the uncertainty of Q-value with neural network ensembles [12], where the consistent Q-value estimates indicate high confidence and can be plausibly used during learning process, even for OOD state-action pairs [13,14]. However, the uncertainty quantification over OOD region highly relies on how neural network generalizes [15].…”

Section: Introductionmentioning

confidence: 99%

See 1 more Smart Citation

Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief

Guo¹,

Shao²,

Geng³

2022

Preprint

View full text Add to dashboard Cite

Model-based offline reinforcement learning (RL) aims to find highly rewarding policy, by leveraging a previously collected static dataset and a dynamics model. While the dynamics model learned through reuse of the static dataset, its generalization ability hopefully promotes policy learning if properly utilized. To that end, several works propose to quantify the uncertainty of predicted dynamics, and explicitly apply it to penalize reward. However, as the dynamics and the reward are intrinsically different factors in context of MDP, characterizing the impact of dynamics uncertainty through reward penalty may incur unexpected tradeoff between model utilization and risk avoidance. In this work, we instead maintain a belief distribution over dynamics, and evaluate/optimize policy through biased sampling from the belief. The sampling procedure, biased towards pessimism, is derived based on an alternating Markov game formulation of offline RL. We formally show that the biased sampling naturally induces an updated dynamics belief with policy-dependent reweighting factor, termed Pessimism-Modulated Dynamics Belief. To improve policy, we devise an iterative regularized policy optimization algorithm for the game, with guarantee of monotonous improvement under certain condition. To make practical, we further devise an offline RL algorithm to approximately find the solution. Empirical results show that the proposed approach achieves state-of-the-art performance on a wide range of benchmark tasks.

show abstract

Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning

Cited by 9 publications

References 6 publications

STEEL: Singularity-aware Reinforcement Learning

STEEL: Singularity-aware Reinforcement Learning

Refined Value-Based Offline RL under Realizability and Partial Coverage

Model-Based Offline Reinforcement Learning with Pessimism-Modulated Dynamics Belief

Contact Info

Product

Resources

About