Monotonic Quantile Network for Worst-Case Offline Reinforcement Learning

Bai, Chenjia; Xiao, Ting; Zhu, Zhigang; Wang, Lingxiao; Zhou, Fan; Garg, Animesh; He, Bin; Liu, Peng; Wang, Zhaoran

doi:10.1109/tnnls.2022.3217189

Cited by 14 publications

(21 citation statements)

References 20 publications

Supporting

Mentioning

Contrasting

Order By: Relevance

“…This limits the applicability of pessimism-based, provably efficient offline RL to practical settings. A very recent work Bai et al (2022) estimates the uncertainty for constructing LCB via the disagreement of bootstrapped Q-functions. However, the uncertainty quantifier is only guaranteed in linear MDPs and must be computed explicitly.…”

Section: Related Workmentioning

confidence: 99%

“…The penalization term R(θ i ; s, φ) discourages overestimation in the value function estimate Q θ i for out-of-distribution (OOD) actions a ∼ π φ (•|s). Our design of R(θ i ; s, φ) is initially inspired by the OOD penalization in Bai et al (2022) that creates a pessimistic pseudo target for the values at OOD actions. Note that we do not need any penalization for OOD actions in our experiment for contextual bandits in Section 6.2.…”

Section: A3 Experiments In D4rl Benchmarkmentioning

confidence: 99%

“…For β, we decrease it from 0.5 to 0.2 by linear decay for the first 50K steps and exponential decay for the remaining steps. For the other hyperparameters of actor-critic training, we fix them the same as in Bai et al (2022). Specifically, the Q-network is the fully connected neural network with three hidden layers all of which has 256 neurons.…”

Section: A3 Experiments In D4rl Benchmarkmentioning

confidence: 99%

“…We compare VIPeR with several state-of-the-art algorithms, including (i) BEAR (Kumar et al, 2019) that use MMD distance to constraint policy to the offline data, (ii) UWAC that improves BEAR using dropout uncertainty, (iii) CQL (Kumar et al, 2020) that minimizes Q-values of OOD actions, (iv) MOPO (Yu et al, 2020) that uses model-based uncertainty via ensemble dynamics, (v) TD3-BC (Fujimoto & Gu, 2021) that uses adaptive behavior cloning, and (vi) PBRL (Bai et al, 2022) that use uncertainty quantification via disagreement of bootstrapped Q-functions. We follow the evaluation protocol in Bai et al (2022). We run our algorithm for five seeds and report the average final evaluation scores with standard deviation.…”

Section: A3 Experiments In D4rl Benchmarkmentioning

confidence: 99%

“…A key challenge in offline RL is to efficiently exploit the given offline dataset to learn an optimal policy in the absence of any further exploration. The dominant approaches to offline RL address this challenge by incorporating uncertainty from the offline dataset into decision-making (Buckman et al, 2021;Jin et al, 2021;Xiao et al, 2021;Nguyen-Tang et al, 2022a;Ghasemipour et al, 2022;Bai et al, 2022). The main component of these uncertainty-aware approaches to offline RL is the pessimism principle, which constrains the learned policy to the offline data and leads to various lower confidence bound (LCB)-based algorithms.…”

Section: Introductionmentioning

confidence: 99%

See 4 more Smart Citations

VIPeR: Provably Efficient Algorithm for Offline RL with Neural Function Approximation

Thanh¹,

Arora²

2023

Preprint

View full text Add to dashboard Cite

We propose a novel algorithm for offline reinforcement learning called Value Iteration with Perturbed Rewards (VIPeR), which amalgamates the pessimism principle with random perturbations of the value function. Most current offline RL algorithms explicitly construct statistical confidence regions to obtain pessimism via lower confidence bounds (LCB), which cannot easily scale to complex problems where a neural network is used to estimate the value functions. Instead, VIPeR implicitly obtains pessimism by simply perturbing the offline data multiple times with carefully-designed i.i.d. Gaussian noises to learn an ensemble of estimated state-action value functions and acting greedily with respect to the minimum of the ensemble. The estimated state-action values are obtained by fitting a parametric model (e.g., neural networks) to the perturbed datasets using gradient descent. As a result, VIPeR only needs O(1) time complexity for action selection, while LCB-based algorithms require at least Ω(K 2 ), where K is the total number of trajectories in the offline data. We also propose a novel data-splitting technique that helps remove a factor involving the log of the covering number in our bound. We prove that VIPeR yields a provable uncertainty quantifier with overparameterized neural networks and enjoys a bound on sub-optimality of Õ(κH 5/2 d/ √ K), where d is the effective dimension, H is the horizon length and κ measures the distributional shift. We corroborate the statistical and computational efficiency of VIPeR with an empirical evaluation on a wide set of synthetic and real-world datasets. To the best of our knowledge, VIPeR is the first algorithm for offline RL that is provably efficient for general Markov decision processes (MDPs) with neural network function approximation.

show abstract

Section: Related Workmentioning

confidence: 99%

Section: A3 Experiments In D4rl Benchmarkmentioning

confidence: 99%

Section: A3 Experiments In D4rl Benchmarkmentioning

confidence: 99%

Section: A3 Experiments In D4rl Benchmarkmentioning

confidence: 99%

Section: Introductionmentioning

confidence: 99%

See 3 more Smart Citations